Sep 18 06:20:12.238486 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 06:20:12.238497 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 06:20:12.238504 ip-10-0-143-147 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 06:20:12.238649 ip-10-0-143-147 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 06:20:22.284567 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 06:20:22.284583 ip-10-0-143-147 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6632a427482545f1b0d808fbcbabec13 --
Sep 18 06:22:00.294425 ip-10-0-143-147 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 06:22:00.715395 ip-10-0-143-147 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 06:22:00.715395 ip-10-0-143-147 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 06:22:00.715395 ip-10-0-143-147 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 06:22:00.715395 ip-10-0-143-147 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 06:22:00.715395 ip-10-0-143-147 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 06:22:00.717815 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.717730    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 06:22:00.721594 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721579    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721596    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721601    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721605    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721608    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721611    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721614    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721617    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721620    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721623    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721626    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721629    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721632    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721635    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.721632 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721638    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721641    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721643    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721646    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721649    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721652    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721655    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721657    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721660    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721662    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721665    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721668    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721670    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721673    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721676    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721679    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721684    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721688    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721690    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.721960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721693    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721696    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721699    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721702    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721704    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721707    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721710    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721712    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721715    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721719    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721722    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721724    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721727    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721731    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721733    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721736    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721739    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721742    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721744    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721747    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.722424 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721749    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721752    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721754    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721757    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721759    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721762    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721764    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721767    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721770    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721772    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721776    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721778    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721781    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721783    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721786    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721788    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721791    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721793    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721796    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.722914 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721799    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721801    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721804    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721806    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721809    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721811    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721814    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721817    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721820    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721823    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721825    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721828    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721830    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721833    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721836    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721838    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721841    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721843    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721846    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721848    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.723419 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721850    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721853    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721856    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721858    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721861    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721863    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.721866    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722546    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722553    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722557    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722560    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722563    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722575    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722578    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722582    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722585    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722588    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722592    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722596    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.723900 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722599    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722603    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722606    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722609    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722619    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722623    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722626    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722629    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722631    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722634    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722636    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722639    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722642    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722644    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722647    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722649    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722653    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722656    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722659    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722661    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.724371 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722664    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722666    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722668    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722671    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722674    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722676    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722680    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722682    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722685    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722687    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722690    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722693    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722696    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722699    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722702    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722705    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722707    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722710    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722712    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722716    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.724864 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722720    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722723    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722727    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722730    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722732    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722735    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722738    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722741    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722744    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722747    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722749    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722752    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722754    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722757    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722759    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722762    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722765    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722767    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722770    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722773    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.725355 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722776    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722779    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722782    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722784    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722787    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722790    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722792    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722795    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722797    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722800    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722802    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722805    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722807    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722810    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722812    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722815    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722817    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722820    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722823    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.725852 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722827    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722830    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722832    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722835    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722838    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722840    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722843    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.722845    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723220    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723225    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723228    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723231    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723235    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723240    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723243    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723246    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723249    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723251    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723254    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723258    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.726325 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723262    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723265    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723268    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723271    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723274    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723277    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723280    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723282    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723285    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723288    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723305    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723308    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723312    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723315    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723318    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723320    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723323    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723325    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723328    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723330    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.726813 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723334    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723336    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723339    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723342    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723344    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723350    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723354    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723357    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723360    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723363    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723366    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723369    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723371    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723374    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723378    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723380    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723383    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723385    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723388    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.727320 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723390    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723393    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723395    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723398    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723400    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723403    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723405    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723408    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723411    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723414    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723416    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723419    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723422    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723425    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723428    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723431    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723434    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723437    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723441    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723445    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.727814 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723447    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723451    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723454    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723458    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723460    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723463    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723466    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723468    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723471    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723473    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723476    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723478    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723481    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723483    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723486    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723488    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723491    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723494    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723496    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.728305 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723499    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723501    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723503    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723506    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723508    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723511    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723514    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723516    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.723519    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723584    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723623    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723632    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723636    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723642    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723645    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723649    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723653    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723656    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723660    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723663    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723666    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723670    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 06:22:00.728765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723673    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723676    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723678    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723681    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723684    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723689    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723692    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723695    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723698    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723701    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723705    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723707    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723711    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723714    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723717    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723720    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723723    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723726    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723729    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723733    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723736    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723741    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723744    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723747    2567 flags.go:64] FLAG: --enable-server="true"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723751    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 06:22:00.729304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723755    2567 flags.go:64] FLAG: --event-burst="100"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723758    2567 flags.go:64] FLAG: --event-qps="50"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723761    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723764    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723767    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723771    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723774    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723776    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723779    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723782    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723785    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723788    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723793    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723796    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723799    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723801    2567 flags.go:64] FLAG: --feature-gates=""
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723805    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723808    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723812    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723815    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723818    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723821    2567 flags.go:64] FLAG: --help="false"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723823    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-143-147.ec2.internal"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723827    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 06:22:00.729966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723830    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723833    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723836    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723839    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723843    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723846    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723849    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723852    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723855    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723858    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723861    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723864    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723867    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723870    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723873    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723876    2567 flags.go:64] FLAG: --lock-file=""
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723878    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723881    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723884    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723889    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723895    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723897    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723900    2567 flags.go:64] FLAG: --logging-format="text"
Sep 18 06:22:00.730596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723903    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723907    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723910    2567 flags.go:64] FLAG: --manifest-url=""
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723913    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723917    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723920    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723924    2567 flags.go:64] FLAG: --max-pods="110"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723927    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723930    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723933    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723936    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723939    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723942    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723946    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723953    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723956    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723959    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723962    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723966    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723971    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723974    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723977    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723980    2567 flags.go:64] FLAG: --port="10250"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723983    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 06:22:00.731160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723986    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03bd3f5a2d53a2a53"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723989    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723992    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723995    2567 flags.go:64] FLAG: --register-node="true"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.723998    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724003    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724006    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724009    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724011    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724015    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724018    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724021    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724024    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724027    2567 flags.go:64] FLAG: --runonce="false"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724029    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724032    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724035    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724038    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724041    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724044    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724047    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724050    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724053    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724056    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724059    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724062    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 06:22:00.731767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724065    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724069    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724072    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724078    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724081    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724083    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724088    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724091    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724093    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724096    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724099    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724103    2567 flags.go:64] FLAG: --v="2"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724107    2567 flags.go:64] FLAG: --version="false"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724111    2567 flags.go:64] FLAG: --vmodule=""
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724115    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724118    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724243    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724248    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724251    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724254    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724257    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724260    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724263    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.732411 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724266    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724268    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724271    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724274    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724276    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724279    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724282    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724285    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724287    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724290    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724292    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724296    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724298    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724302    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724305    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724308    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724311    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724313    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724316    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724318    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.732960 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724322    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724325    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724329    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724332    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724335    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724337    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724340    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724343    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724346    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724349    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724352    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724354    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724357    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724360    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724362    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724365    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724367    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724370    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724373    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.733461 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724376    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724379    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724382    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724385    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724387    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724391    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724394    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724397    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724399    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724402    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724404    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724407    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724409    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724413    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724415    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724418    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724421    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724425    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724429    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724433    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.733917 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724437    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724439    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724442    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724445    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724447    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724450    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724453    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724455    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724458    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724460    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724464    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724467    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724470    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724473    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724475    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724478    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724481    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724485    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724489    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.734410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724493    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724496    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724499    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724502    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724505    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724508    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724512    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724515    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724517    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724520    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724523    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724525    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724528    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.724530    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.724536    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 06:22:00.734940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.731776    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.731791    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.731809    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.731816    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731872    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731876    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731880    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731884    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731888    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731892    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731896    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731899    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731902    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731905    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731908    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731911    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731913    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731916    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731919    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.735336 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731922    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731924    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731927    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731930    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731932    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731934    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731938    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731942    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731945    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731948    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731951    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731953    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731956    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731958    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731963    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731965    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731968    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731971    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731973    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.735894 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731977    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731981    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731984    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731987    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731990    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731993    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731995    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.731998    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732000    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732003    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732005    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732008    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732010    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732013    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732015    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732018    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732021    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732023    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732026    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732028    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.736400 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732031    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732034    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732036    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732039    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732041    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732044    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732047    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732050    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732053    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732056    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732059    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732061    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732064    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732066    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732069    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732071    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732074    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732076    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732079    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732081    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.736911 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732084    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732087    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732089    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732092    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732095    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732098    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732100    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732103    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732105    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732108    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732110    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732113    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732115    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732118    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732120    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732123    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732125    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732128    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732131    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732148    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.737407 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732152    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732157    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732161    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732166    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732169    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.732174    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732285    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732290    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732293    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732296    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732299    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732302    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732304    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732307    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 06:22:00.737969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732310    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732313    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732316    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732318    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732321    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732324    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732327    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732329    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732332    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732336    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732340    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732343    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732346    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732349    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732353    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732355    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732358    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732360    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732363    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 06:22:00.738508 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732366    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732370    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732372    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732375    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732378    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732380    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732383    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732385    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732388    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732390    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732393    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732395    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732398    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732400    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732403    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732406    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732409    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732412    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732415    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732417    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 06:22:00.739282 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732420    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732423    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732426    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732428    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732431    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732433    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732436    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732439    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732442    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732444    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732447    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732449    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732452    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732455    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732458    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732460    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732463    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732466    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732468    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732471    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 06:22:00.740020 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732473    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732476    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732479    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732481    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732484    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732486    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732489    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732491    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732494    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732496    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732499    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732501    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732504    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732507    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732509    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732512    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732514    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732517    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732519    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732522    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 06:22:00.740688 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732525    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732527    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732530    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732532    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732535    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732538    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732541    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732545    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732548    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732551    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732554    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:00.732557    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.732562    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.732660    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 06:22:00.741299 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.734554    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 06:22:00.741685 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.735609    2567 server.go:1019] "Starting client certificate rotation"
Sep 18 06:22:00.741685 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.735728    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 06:22:00.741685 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.735772    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 06:22:00.741685 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.738337    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 06:22:00.741685 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.741339    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 06:22:00.753610 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.753587    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 18 06:22:00.757112 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.757094    2567 log.go:25] "Validated CRI v1 image API"
Sep 18 06:22:00.759278 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.759260    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 06:22:00.763332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.763308    2567 fs.go:135] Filesystem UUIDs: map[572bd7e6-09b4-4807-989c-3fea716cd064:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8a9c2806-63c0-418c-a277-68c7be3a7a1c:/dev/nvme0n1p4]
Sep 18 06:22:00.763406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.763331    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 06:22:00.766304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.766287    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 06:22:00.768321 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.768218    2567 manager.go:216] Machine: {Timestamp:2026-09-18 06:22:00.766833803 +0000 UTC m=+0.350219446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098825 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b7fdc412d88852e497dafa8693f3 SystemUUID:ec22b7fd-c412-d888-52e4-97dafa8693f3 BootID:6632a427-4825-45f1-b0d8-08fbcbabec13 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:c0:80:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:c0:80:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:11:03:a4:45:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 06:22:00.768321 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.768313    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 06:22:00.768428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.768383    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 06:22:00.771772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.771751    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 06:22:00.771901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.771774    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 06:22:00.773025 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.773015    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 06:22:00.773062 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.773027    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 06:22:00.773062 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.773034    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 06:22:00.773652 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.773643    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 06:22:00.775173 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.775163    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 06:22:00.778375 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778364    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 06:22:00.778474 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778465    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 06:22:00.778503 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778493    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 06:22:00.778536 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778505    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 06:22:00.778536 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778520    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 06:22:00.778536 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778529    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 18 06:22:00.778536 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778537    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 06:22:00.778910 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778897    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 06:22:00.778951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.778925    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 06:22:00.782082 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.782061    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 06:22:00.784816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.784781    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 06:22:00.784878 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.784832    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 06:22:00.786283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786271    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786287    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786293    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786298    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786307    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786312    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786325    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786330    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786337    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 06:22:00.786337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786342    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 06:22:00.786750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786351    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 06:22:00.786750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.786356    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 06:22:00.787202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.787190    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 06:22:00.787202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.787201    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 06:22:00.789041 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.789018    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 06:22:00.789096 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.789047    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 06:22:00.789207 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.789187    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 06:22:00.791245 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.791080    2567 server.go:1268] "Started kubelet"
Sep 18 06:22:00.791311 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.791126    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 06:22:00.791399 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.791180    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 06:22:00.791515 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.791420    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 06:22:00.792022 ip-10-0-143-147 systemd[1]: Started Kubernetes Kubelet.
Sep 18 06:22:00.793051 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.793033    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 06:22:00.794532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.793717    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 06:22:00.797651 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.797634    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 06:22:00.800583 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.800565    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 06:22:00.800662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.800588    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 06:22:00.800763 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.800726    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:00.801259 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801241    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 06:22:00.801259 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801250    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 06:22:00.801376 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801267    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 06:22:00.801376 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801336    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 06:22:00.801376 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801346    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 06:22:00.801519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801507    2567 factory.go:55] Registering systemd factory
Sep 18 06:22:00.801564 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801521    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 18 06:22:00.801731 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801717    2567 factory.go:153] Registering CRI-O factory
Sep 18 06:22:00.801796 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801735    2567 factory.go:223] Registration of the crio container factory successfully
Sep 18 06:22:00.801796 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801779    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 06:22:00.801796 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801795    2567 factory.go:103] Registering Raw factory
Sep 18 06:22:00.801941 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.801810    2567 manager.go:1196] Started watching for new ooms in manager
Sep 18 06:22:00.802273 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.802260    2567 manager.go:349] Starting recovery of all containers
Sep 18 06:22:00.807091 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.807064    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 06:22:00.807217 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.807179    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 06:22:00.808351 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.807157    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18d6560e2bd1cc45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:ip-10-0-143-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-09-18 06:22:00.791051333 +0000 UTC m=+0.374436976,LastTimestamp:2026-09-18 06:22:00.791051333 +0000 UTC m=+0.374436976,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}"
Sep 18 06:22:00.809385 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.809342    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 06:22:00.812372 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.812351    2567 manager.go:354] Recovery completed
Sep 18 06:22:00.814880 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.814857    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxsh6"
Sep 18 06:22:00.818491 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.818478    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:00.819939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.819923    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxsh6"
Sep 18 06:22:00.820238 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820222    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:00.820287 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820251    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:00.820287 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820261    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:00.820763 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820750    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 06:22:00.820801 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820762    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 06:22:00.820801 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820778    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 06:22:00.820864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.820763    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:00.821538 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.821518    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:00.821603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.821547    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:00.821603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.821562    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:00.821920 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.821847    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18d6560e2d8f2a20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:ip-10-0-143-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-09-18 06:22:00.82023888 +0000 UTC m=+0.403624524,LastTimestamp:2026-09-18 06:22:00.82023888 +0000 UTC m=+0.403624524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}"
Sep 18 06:22:00.823289 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.823276    2567 policy_none.go:49] "None policy: Start"
Sep 18 06:22:00.823352 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.823292    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 06:22:00.823352 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.823303    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 06:22:00.824881 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.824870    2567 policy_none.go:47] "Start"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854414    2567 manager.go:337] "Starting Device Plugin manager"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.854455    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854464    2567 server.go:87] "Starting device plugin registration server"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854784    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854796    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854890    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854965    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.854973    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.855431    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 06:22:00.875909 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.855474    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:00.902716 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.902691    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 06:22:00.902825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.902741    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 06:22:00.902825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.902761    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 06:22:00.902825 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.902794    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 06:22:00.904765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.904745    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 06:22:00.955412 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.955385    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:00.956238 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.956223    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:00.956306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.956254    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:00.956306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.956266    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:00.956306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.956288    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-147.ec2.internal"
Sep 18 06:22:00.966681 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:00.966594    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-147.ec2.internal"
Sep 18 06:22:00.966681 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.966628    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-147.ec2.internal\": node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:00.989887 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:00.989860    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.003089 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.003066    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"]
Sep 18 06:22:01.003172 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.003163    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:01.004647 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.004631    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:01.004741 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.004666    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:01.004741 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.004680    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:01.006049 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006033    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:01.006170 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.006217 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006185    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:01.006765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006749    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:01.006829 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006774    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:01.006829 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006798    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:01.006829 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006812    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:01.006960 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006778    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:01.006960 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.006867    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:01.008998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.008980    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.009073 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.009011    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 06:22:01.009705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.009685    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 06:22:01.009790 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.009709    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 06:22:01.009790 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.009723    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 06:22:01.027730 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.027710    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.031004 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.030986    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.090841 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.090821    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.191323 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.191284    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.202617 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.202586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65769deb7fc6587d12564f9ff3479c19-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"65769deb7fc6587d12564f9ff3479c19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.202692 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.202621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.202692 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.202639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.292043 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.291979    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.303338 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.303404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303342    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65769deb7fc6587d12564f9ff3479c19-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"65769deb7fc6587d12564f9ff3479c19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.303404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.303463 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303400    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.303463 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303403    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c53452cc9bc905a4f008dcb7e85cd17e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"c53452cc9bc905a4f008dcb7e85cd17e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.303463 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.303430    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/65769deb7fc6587d12564f9ff3479c19-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"65769deb7fc6587d12564f9ff3479c19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.331427 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.331392    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.334782 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.334765    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:01.392511 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.392471    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.493036 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.492989    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.593655 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.593572    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.694238 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.694187    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.736966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.736932    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 06:22:01.737408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.737103    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 06:22:01.795036 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.795011    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.798510 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.798489    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 06:22:01.827261 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.826573    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 06:17:01 +0000 UTC" deadline="2028-04-04 05:13:08.75428786 +0000 UTC"
Sep 18 06:22:01.827414 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.827267    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13534h51m6.927029032s"
Sep 18 06:22:01.895522 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.895435    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:01.921716 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:01.921499    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53452cc9bc905a4f008dcb7e85cd17e.slice/crio-88af1e7d4592634463219ed875da43a4b72b387e450fa61b0aeab6fc97dca849 WatchSource:0}: Error finding container 88af1e7d4592634463219ed875da43a4b72b387e450fa61b0aeab6fc97dca849: Status 404 returned error can't find the container with id 88af1e7d4592634463219ed875da43a4b72b387e450fa61b0aeab6fc97dca849
Sep 18 06:22:01.925776 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:01.925760    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:22:01.961116 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:01.961079    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65769deb7fc6587d12564f9ff3479c19.slice/crio-12bcc6b199a0f2a7eb09fcc64af2d5a5d616950155fe0611a517636df9951d1c WatchSource:0}: Error finding container 12bcc6b199a0f2a7eb09fcc64af2d5a5d616950155fe0611a517636df9951d1c: Status 404 returned error can't find the container with id 12bcc6b199a0f2a7eb09fcc64af2d5a5d616950155fe0611a517636df9951d1c
Sep 18 06:22:01.996331 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:01.996283    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:02.099491 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.099433    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:02.201367 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.201262    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:02.301398 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.301354    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found"
Sep 18 06:22:02.304744 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.304728    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 06:22:02.348864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.348831    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 06:22:02.403454 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.403409    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:02.416444 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.416422    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 06:22:02.417520 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.417506    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"
Sep 18 06:22:02.425655 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.425638    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 06:22:02.548988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.548897    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 06:22:02.780174 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.780125    2567 apiserver.go:52] "Watching apiserver"
Sep 18 06:22:02.790496 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.790467    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 06:22:02.791094 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.791037    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-k6qw9","kube-system/konnectivity-agent-vbqkl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j","openshift-dns/node-resolver-lflsh","openshift-multus/multus-additional-cni-plugins-n75xx","openshift-multus/multus-jbvbm","openshift-multus/network-metrics-daemon-fwqdd","openshift-ovn-kubernetes/ovnkube-node-p645l","kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal","openshift-cluster-node-tuning-operator/tuned-69qzh","openshift-image-registry/node-ca-fqclk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal","openshift-network-diagnostics/network-check-target-rk2dg","openshift-network-operator/iptables-alerter-vlxrj"]
Sep 18 06:22:02.793074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.793052    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.793192 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.793145    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:02.794248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.794229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:02.794346 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.794288    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:02.795858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.795837    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.798528 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.798504    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 06:22:02.798948 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.798927    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xbl5k\""
Sep 18 06:22:02.799356 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.799309    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.799443 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.799380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.799779 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.799761    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.801665 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.801583    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 06:22:02.801665 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.801654    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.802179 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.802099    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.802533 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.802515    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.802666 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.802643    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t779k\""
Sep 18 06:22:02.803077 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.803056    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804528    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804549    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804794    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xbvln\""
Sep 18 06:22:02.805333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.804846    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:02.805985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.805655    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 06:22:02.805985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.805879    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 06:22:02.805985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.805909    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nrw2l\""
Sep 18 06:22:02.806239 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.806206    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:02.806301 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.806272    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:02.807456 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.807314    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.807456 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.807324    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.807606 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.807476    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 06:22:02.807745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.807725    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2mkml\""
Sep 18 06:22:02.807805 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.807766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:02.809345 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.809327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.811441 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.811411    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:02.813937 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.813911    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cqdmx\""
Sep 18 06:22:02.814186 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814169    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 06:22:02.814340 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814319    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jm5hn\""
Sep 18 06:22:02.814443 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814426    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 06:22:02.814866 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814649    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.814866 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814774    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.814866 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.814796    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5zhk\""
Sep 18 06:22:02.815122 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815039    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.815196 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815160    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x64\" (UniqueName: \"kubernetes.io/projected/d5ad6271-e63e-4598-a728-7d268afb74c9-kube-api-access-z8x64\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815434    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-multus\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815464    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-cnibin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815508 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-cni-binary-copy\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-registration-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.815842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815664    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-bin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-multus-certs\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.815842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.815842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-system-cni-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mc6q\" (UniqueName: \"kubernetes.io/projected/b395dace-429c-4e52-9e3e-ca1328cc8320-kube-api-access-5mc6q\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-netns\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815948    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.815979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxjn\" (UniqueName: \"kubernetes.io/projected/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-kube-api-access-9vxjn\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816011    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-kubelet\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-socket-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.816093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816070    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-tmp-dir\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816132    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rhd\" (UniqueName: \"kubernetes.io/projected/7f327502-c45b-4c98-802a-a4ad54e6fbfa-kube-api-access-b7rhd\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-hosts-file\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816276    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-system-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-sys-fs\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-cnibin\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816395    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-binary-copy\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816419    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-hostroot\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-multus-daemon-config\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816481    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-etc-kubernetes\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcl9\" (UniqueName: \"kubernetes.io/projected/a3644ffb-4d40-476d-ace4-5af038423533-kube-api-access-gtcl9\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816544    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-device-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816570    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-socket-dir-parent\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816608    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-k8s-cni-cncf-io\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.816977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.816638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-conf-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.818457 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.818380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.819318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.818713    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.819512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.819495    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 06:22:02.821401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.821377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gc69j\""
Sep 18 06:22:02.821680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.821659    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 06:22:02.822020 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.822000    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 06:22:02.822116 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.822073    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 06:22:02.822298 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.822280    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 06:22:02.822378 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.822340    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 06:22:02.823143 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.823120    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 06:22:02.848747 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.848721    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvq4b"
Sep 18 06:22:02.871383 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.871307    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvq4b"
Sep 18 06:22:02.902774 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.902679    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 06:22:02.909425 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.909365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"65769deb7fc6587d12564f9ff3479c19","Type":"ContainerStarted","Data":"12bcc6b199a0f2a7eb09fcc64af2d5a5d616950155fe0611a517636df9951d1c"}
Sep 18 06:22:02.912094 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.912047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"c53452cc9bc905a4f008dcb7e85cd17e","Type":"ContainerStarted","Data":"88af1e7d4592634463219ed875da43a4b72b387e450fa61b0aeab6fc97dca849"}
Sep 18 06:22:02.916797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:02.916921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-slash\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.916921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-config\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.916921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916858    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-systemd\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.916921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x64\" (UniqueName: \"kubernetes.io/projected/d5ad6271-e63e-4598-a728-7d268afb74c9-kube-api-access-z8x64\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.916921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916913    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-cnibin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-bin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-log-socket\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.916986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-kubernetes\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917010    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-tmp\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-system-cni-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs5lr\" (UniqueName: \"kubernetes.io/projected/cad7595d-601f-47c1-9ec6-758e058999d1-kube-api-access-rs5lr\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-systemd-units\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-kubelet\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917144    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-kubelet\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917169    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-socket-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.917223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917198    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-cnibin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-tmp-dir\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-modprobe-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917359    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-var-lib-kubelet\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917415    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-system-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-sys-fs\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917499    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-cnibin\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-binary-copy\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-multus-daemon-config\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f1513d-e978-4051-9a08-56ad33a5e160-host\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-system-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-netns\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-script-lib\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.917997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-sys-fs\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917739    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-systemd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.917739    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-var-lib-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.917840    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:03.417781504 +0000 UTC m=+3.001167137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-bin\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-system-cni-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918005    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-kubelet\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918092    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-tmp-dir\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918205    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-socket-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.917592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-cnibin\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918391    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-binary-copy\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-device-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-socket-dir-parent\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918513    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-k8s-cni-cncf-io\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-conf-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918567    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kp75\" (UniqueName: \"kubernetes.io/projected/f7f1513d-e978-4051-9a08-56ad33a5e160-kube-api-access-6kp75\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:02.918895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918587    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-ovn\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918634    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-conf-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918660    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-multus\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918720    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-cni-dir\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-netd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-device-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0959cf8e-4192-40bb-9351-5753f6d0e440-konnectivity-ca\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-multus-socket-dir-parent\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-multus-daemon-config\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918838    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-k8s-cni-cncf-io\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918831    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-var-lib-cni-multus\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-conf\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918938    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-cni-binary-copy\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.919860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.918999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-registration-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919029    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-multus-certs\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919055    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28597235-7299-4269-8e10-45bb68e87f84-ovn-node-metrics-cert\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919100    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-registration-dir\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgct\" (UniqueName: \"kubernetes.io/projected/28597235-7299-4269-8e10-45bb68e87f84-kube-api-access-qxgct\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysconfig\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b395dace-429c-4e52-9e3e-ca1328cc8320-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-lib-modules\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-multus-certs\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mc6q\" (UniqueName: \"kubernetes.io/projected/b395dace-429c-4e52-9e3e-ca1328cc8320-kube-api-access-5mc6q\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-netns\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919443    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-env-overrides\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ad6271-e63e-4598-a728-7d268afb74c9-etc-selinux\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-run\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919497    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3644ffb-4d40-476d-ace4-5af038423533-cni-binary-copy\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.920714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-host\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-host-run-netns\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-tuned\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxjn\" (UniqueName: \"kubernetes.io/projected/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-kube-api-access-9vxjn\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919640    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-kubelet-config\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-sys\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bq6\" (UniqueName: \"kubernetes.io/projected/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-kube-api-access-f2bq6\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rhd\" (UniqueName: \"kubernetes.io/projected/7f327502-c45b-4c98-802a-a4ad54e6fbfa-kube-api-access-b7rhd\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-hosts-file\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919799    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919823    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7f1513d-e978-4051-9a08-56ad33a5e160-serviceca\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-hosts-file\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-etc-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0959cf8e-4192-40bb-9351-5753f6d0e440-agent-certs\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.919987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b395dace-429c-4e52-9e3e-ca1328cc8320-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:02.921579 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-hostroot\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-etc-kubernetes\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920057    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcl9\" (UniqueName: \"kubernetes.io/projected/a3644ffb-4d40-476d-ace4-5af038423533-kube-api-access-gtcl9\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920112    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-etc-kubernetes\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cad7595d-601f-47c1-9ec6-758e058999d1-iptables-alerter-script\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920205    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3644ffb-4d40-476d-ace4-5af038423533-hostroot\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad7595d-601f-47c1-9ec6-758e058999d1-host-slash\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-dbus\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-node-log\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.922477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.920341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-bin\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:02.923687 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.923591    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 06:22:02.925733 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.925593    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:02.925733 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.925619    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:02.925733 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.925634    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:02.925733 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:02.925702    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:03.425684295 +0000 UTC m=+3.009069946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:02.928927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.928848    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x64\" (UniqueName: \"kubernetes.io/projected/d5ad6271-e63e-4598-a728-7d268afb74c9-kube-api-access-z8x64\") pod \"aws-ebs-csi-driver-node-qns5j\" (UID: \"d5ad6271-e63e-4598-a728-7d268afb74c9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:02.931631 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.931582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxjn\" (UniqueName: \"kubernetes.io/projected/6d5ac4a2-e29f-4c7f-b9db-d8c021a09487-kube-api-access-9vxjn\") pod \"node-resolver-lflsh\" (UID: \"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487\") " pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:02.932337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.932295    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rhd\" (UniqueName: \"kubernetes.io/projected/7f327502-c45b-4c98-802a-a4ad54e6fbfa-kube-api-access-b7rhd\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:02.932927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.932898    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcl9\" (UniqueName: \"kubernetes.io/projected/a3644ffb-4d40-476d-ace4-5af038423533-kube-api-access-gtcl9\") pod \"multus-jbvbm\" (UID: \"a3644ffb-4d40-476d-ace4-5af038423533\") " pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:02.934549 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:02.934505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mc6q\" (UniqueName: \"kubernetes.io/projected/b395dace-429c-4e52-9e3e-ca1328cc8320-kube-api-access-5mc6q\") pod \"multus-additional-cni-plugins-n75xx\" (UID: \"b395dace-429c-4e52-9e3e-ca1328cc8320\") " pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:03.020792 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-run\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-host\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-tuned\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-kubelet-config\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-sys\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bq6\" (UniqueName: \"kubernetes.io/projected/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-kube-api-access-f2bq6\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.020966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7f1513d-e978-4051-9a08-56ad33a5e160-serviceca\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.020992    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-etc-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0959cf8e-4192-40bb-9351-5753f6d0e440-agent-certs\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cad7595d-601f-47c1-9ec6-758e058999d1-iptables-alerter-script\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad7595d-601f-47c1-9ec6-758e058999d1-host-slash\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-dbus\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021115    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-node-log\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-bin\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-slash\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-config\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-systemd\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021272    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cad7595d-601f-47c1-9ec6-758e058999d1-host-slash\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-node-log\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-log-socket\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-kubelet-config\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021348    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-log-socket\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-kubernetes\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-tmp\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021396    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-host\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs5lr\" (UniqueName: \"kubernetes.io/projected/cad7595d-601f-47c1-9ec6-758e058999d1-kube-api-access-rs5lr\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-dbus\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-bin\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-systemd-units\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-kubelet\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-kubernetes\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.021627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-modprobe-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-var-lib-kubelet\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021597    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f1513d-e978-4051-9a08-56ad33a5e160-host\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021630    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-netns\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021681    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-netns\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-sys\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f7f1513d-e978-4051-9a08-56ad33a5e160-serviceca\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-slash\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021771    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.021795    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-kubelet\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.021859    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:03.521839047 +0000 UTC m=+3.105224686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-modprobe-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.021997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-script-lib\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-systemd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.022639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cad7595d-601f-47c1-9ec6-758e058999d1-iptables-alerter-script\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-var-lib-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022083    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f1513d-e978-4051-9a08-56ad33a5e160-host\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-var-lib-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kp75\" (UniqueName: \"kubernetes.io/projected/f7f1513d-e978-4051-9a08-56ad33a5e160-kube-api-access-6kp75\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-ovn\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-systemd\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022166    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-netd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0959cf8e-4192-40bb-9351-5753f6d0e440-konnectivity-ca\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-conf\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022265    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-systemd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022308    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-config\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022318    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-run-ovn\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28597235-7299-4269-8e10-45bb68e87f84-ovn-node-metrics-cert\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-cni-netd\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022348    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-etc-openvswitch\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.023581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022369    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-run\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-systemd-units\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgct\" (UniqueName: \"kubernetes.io/projected/28597235-7299-4269-8e10-45bb68e87f84-kube-api-access-qxgct\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022035    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-var-lib-kubelet\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-d\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysctl-conf\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysconfig\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-lib-modules\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-ovnkube-script-lib\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-env-overrides\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28597235-7299-4269-8e10-45bb68e87f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-lib-modules\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-sysconfig\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.022850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0959cf8e-4192-40bb-9351-5753f6d0e440-konnectivity-ca\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:03.024523 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.023290    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28597235-7299-4269-8e10-45bb68e87f84-env-overrides\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.025598 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.025576    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0959cf8e-4192-40bb-9351-5753f6d0e440-agent-certs\") pod \"konnectivity-agent-vbqkl\" (UID: \"0959cf8e-4192-40bb-9351-5753f6d0e440\") " pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:03.025772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.025717    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-tmp\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.026010 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.025973    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-etc-tuned\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.027195 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.027138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28597235-7299-4269-8e10-45bb68e87f84-ovn-node-metrics-cert\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.030979 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.030935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kp75\" (UniqueName: \"kubernetes.io/projected/f7f1513d-e978-4051-9a08-56ad33a5e160-kube-api-access-6kp75\") pod \"node-ca-fqclk\" (UID: \"f7f1513d-e978-4051-9a08-56ad33a5e160\") " pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.031127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.031103    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs5lr\" (UniqueName: \"kubernetes.io/projected/cad7595d-601f-47c1-9ec6-758e058999d1-kube-api-access-rs5lr\") pod \"iptables-alerter-vlxrj\" (UID: \"cad7595d-601f-47c1-9ec6-758e058999d1\") " pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.031569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.031547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bq6\" (UniqueName: \"kubernetes.io/projected/2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b-kube-api-access-f2bq6\") pod \"tuned-69qzh\" (UID: \"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b\") " pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.032663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.032645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgct\" (UniqueName: \"kubernetes.io/projected/28597235-7299-4269-8e10-45bb68e87f84-kube-api-access-qxgct\") pod \"ovnkube-node-p645l\" (UID: \"28597235-7299-4269-8e10-45bb68e87f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.109626 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.109535    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j"
Sep 18 06:22:03.123728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.123691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lflsh"
Sep 18 06:22:03.136986 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.136955    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n75xx"
Sep 18 06:22:03.144336 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.144306    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jbvbm"
Sep 18 06:22:03.153344 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.153312    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vlxrj"
Sep 18 06:22:03.162050 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.162028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:03.169966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.169948    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69qzh"
Sep 18 06:22:03.179304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.179286    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fqclk"
Sep 18 06:22:03.188066 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.188047    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:03.426332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.426232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:03.426332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.426322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426447    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426459    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426486    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426500    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426516    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:04.426494391 +0000 UTC m=+4.009880042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:03.426572 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.426558    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:04.426540704 +0000 UTC m=+4.009926339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:03.527716 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.527673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:03.527924 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.527838    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:03.527924 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:03.527911    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:04.527889264 +0000 UTC m=+4.111274901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:03.873926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.873834    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 06:17:03 +0000 UTC" deadline="2028-03-22 07:10:43.179022766 +0000 UTC"
Sep 18 06:22:03.873926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:03.873874    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13224h48m39.305153178s"
Sep 18 06:22:04.149971 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.149939    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f1513d_e978_4051_9a08_56ad33a5e160.slice/crio-af6b30a82239e96726b50d3704ebfea4aca1431cb492eff01412641ce7cc8d94 WatchSource:0}: Error finding container af6b30a82239e96726b50d3704ebfea4aca1431cb492eff01412641ce7cc8d94: Status 404 returned error can't find the container with id af6b30a82239e96726b50d3704ebfea4aca1431cb492eff01412641ce7cc8d94
Sep 18 06:22:04.151084 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.151040    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfbb8e3_a8aa_44f2_9e50_82aac7402a0b.slice/crio-8f542f5cd5dadbdcd5f3fd810237250f06185f78036478637c3e7dbc0ab653ec WatchSource:0}: Error finding container 8f542f5cd5dadbdcd5f3fd810237250f06185f78036478637c3e7dbc0ab653ec: Status 404 returned error can't find the container with id 8f542f5cd5dadbdcd5f3fd810237250f06185f78036478637c3e7dbc0ab653ec
Sep 18 06:22:04.158778 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.158751    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3644ffb_4d40_476d_ace4_5af038423533.slice/crio-70423be096f73bdfa26aa64e2f1140cc5e4295e5b24eb07c82e57e561a47be19 WatchSource:0}: Error finding container 70423be096f73bdfa26aa64e2f1140cc5e4295e5b24eb07c82e57e561a47be19: Status 404 returned error can't find the container with id 70423be096f73bdfa26aa64e2f1140cc5e4295e5b24eb07c82e57e561a47be19
Sep 18 06:22:04.159582 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.159549    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28597235_7299_4269_8e10_45bb68e87f84.slice/crio-1655129f94a9c55c9fc2ae46bdf4e47333074acfdb8f1fbdae4be6c3355e5ead WatchSource:0}: Error finding container 1655129f94a9c55c9fc2ae46bdf4e47333074acfdb8f1fbdae4be6c3355e5ead: Status 404 returned error can't find the container with id 1655129f94a9c55c9fc2ae46bdf4e47333074acfdb8f1fbdae4be6c3355e5ead
Sep 18 06:22:04.161273 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.161217    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb395dace_429c_4e52_9e3e_ca1328cc8320.slice/crio-295a0858f785117e3054c2cf5bb0320267a2021cc22e02322061591fb814996d WatchSource:0}: Error finding container 295a0858f785117e3054c2cf5bb0320267a2021cc22e02322061591fb814996d: Status 404 returned error can't find the container with id 295a0858f785117e3054c2cf5bb0320267a2021cc22e02322061591fb814996d
Sep 18 06:22:04.162359 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:04.161935    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0959cf8e_4192_40bb_9351_5753f6d0e440.slice/crio-cf19c421bb03f5cd63249d4b1120e32fedad6bbb416c9bea943e5526c7a688c0 WatchSource:0}: Error finding container cf19c421bb03f5cd63249d4b1120e32fedad6bbb416c9bea943e5526c7a688c0: Status 404 returned error can't find the container with id cf19c421bb03f5cd63249d4b1120e32fedad6bbb416c9bea943e5526c7a688c0
Sep 18 06:22:04.435591 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.435556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:04.435688 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.435619    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:04.435727 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435702    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:04.435762 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435736    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:04.435762 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435756    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:04.435828 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435764    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:06.435746888 +0000 UTC m=+6.019132544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:04.435828 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435767    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:04.435828 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.435813    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:06.435798397 +0000 UTC m=+6.019184030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:04.537597 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.537563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:04.537782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.537691    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:04.537782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.537754    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:06.537739409 +0000 UTC m=+6.121125056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:04.875531 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.875376    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 06:17:03 +0000 UTC" deadline="2028-03-31 18:13:42.559748222 +0000 UTC"
Sep 18 06:22:04.875531 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.875442    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13451h51m37.684312165s"
Sep 18 06:22:04.906308 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.906271    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:04.906479 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.906416    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:04.906869 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.906844    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:04.906987 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.906967    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:04.907050 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.907037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:04.907137 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:04.907116    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:04.919360 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.918870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fqclk" event={"ID":"f7f1513d-e978-4051-9a08-56ad33a5e160","Type":"ContainerStarted","Data":"af6b30a82239e96726b50d3704ebfea4aca1431cb492eff01412641ce7cc8d94"}
Sep 18 06:22:04.926871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.926833    2567 generic.go:356] "Generic (PLEG): container finished" podID="c53452cc9bc905a4f008dcb7e85cd17e" containerID="af20fea42838131e1ad91ccf46ba17d8ea157532138f20af96a34d8471e4bad8" exitCode=0
Sep 18 06:22:04.927018 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.926929    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"c53452cc9bc905a4f008dcb7e85cd17e","Type":"ContainerDied","Data":"af20fea42838131e1ad91ccf46ba17d8ea157532138f20af96a34d8471e4bad8"}
Sep 18 06:22:04.951406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.951297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j" event={"ID":"d5ad6271-e63e-4598-a728-7d268afb74c9","Type":"ContainerStarted","Data":"0739366f8181493843a10859a9ead252662e2bd19607535b8e969bacf3bc4eb9"}
Sep 18 06:22:04.960120 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.960084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbqkl" event={"ID":"0959cf8e-4192-40bb-9351-5753f6d0e440","Type":"ContainerStarted","Data":"cf19c421bb03f5cd63249d4b1120e32fedad6bbb416c9bea943e5526c7a688c0"}
Sep 18 06:22:04.964192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.964135    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbvbm" event={"ID":"a3644ffb-4d40-476d-ace4-5af038423533","Type":"ContainerStarted","Data":"70423be096f73bdfa26aa64e2f1140cc5e4295e5b24eb07c82e57e561a47be19"}
Sep 18 06:22:04.983000 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.982963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"65769deb7fc6587d12564f9ff3479c19","Type":"ContainerStarted","Data":"ed10e5f927a842aceeb3aecb726a8fe127f12cafbe73a7025c4509f0fea4832e"}
Sep 18 06:22:04.997621 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:04.997557    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" podStartSLOduration=2.997535894 podStartE2EDuration="2.997535894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:04.996869369 +0000 UTC m=+4.580255028" watchObservedRunningTime="2026-09-18 06:22:04.997535894 +0000 UTC m=+4.580921554"
Sep 18 06:22:05.004272 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:05.004237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vlxrj" event={"ID":"cad7595d-601f-47c1-9ec6-758e058999d1","Type":"ContainerStarted","Data":"a81c8474fd4a30d878dc115bff153d891bcb55f1b090a27b3debf5853b77fef2"}
Sep 18 06:22:05.027900 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:05.027865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lflsh" event={"ID":"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487","Type":"ContainerStarted","Data":"c1c48e059be074017834eb63543020d90f706e6e5fcf085af23d173634e8fdde"}
Sep 18 06:22:05.036963 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:05.036927    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerStarted","Data":"295a0858f785117e3054c2cf5bb0320267a2021cc22e02322061591fb814996d"}
Sep 18 06:22:05.060450 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:05.060411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"1655129f94a9c55c9fc2ae46bdf4e47333074acfdb8f1fbdae4be6c3355e5ead"}
Sep 18 06:22:05.071529 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:05.071487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69qzh" event={"ID":"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b","Type":"ContainerStarted","Data":"8f542f5cd5dadbdcd5f3fd810237250f06185f78036478637c3e7dbc0ab653ec"}
Sep 18 06:22:06.086313 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.086267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"c53452cc9bc905a4f008dcb7e85cd17e","Type":"ContainerStarted","Data":"094d4027b011d89299f7bbe4a9ba2a93705de0e97164daf911c0bb18f9d0943b"}
Sep 18 06:22:06.451813 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.451763    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:06.451988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.451851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:06.451988 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.451942    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:06.452262 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.452008    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:06.452262 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.452031    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:06.452262 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.452047    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:06.452262 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.452030    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:10.452008294 +0000 UTC m=+10.035393943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:06.452262 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.452131    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:10.452105034 +0000 UTC m=+10.035490682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:06.553124 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.553083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:06.553335 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.553263    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:06.553335 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.553334    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:10.553314679 +0000 UTC m=+10.136700319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:06.905029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.904950    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:06.905198 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.905080    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:06.905530 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.905503    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:06.905658 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.905609    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:06.905658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:06.905641    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:06.905779 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:06.905734    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:08.903940    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:08.903953    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:08.904091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:08.904089    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:08.904175    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:08.904468 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:08.904271    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:10.494606 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.494568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.494658    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494775    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494783    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494810    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494825    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494837    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:18.494820825 +0000 UTC m=+18.078206480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:10.495022 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.494883    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:18.494865507 +0000 UTC m=+18.078251157 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:10.595904 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.595859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:10.596093 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.596061    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:10.596185 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.596149    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:18.596112271 +0000 UTC m=+18.179497921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:10.904720 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.904611    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:10.904867 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.904718    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:10.905429 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.905087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:10.905429 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.905216    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:10.905429 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:10.905274    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:10.905429 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:10.905342    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:12.903481 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:12.903447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:12.903897 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:12.903590    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:12.903897 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:12.903451    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:12.903897 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:12.903447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:12.903897 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:12.903700    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:12.903897 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:12.903778    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:14.903497 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:14.903465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:14.903497 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:14.903480    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:14.903497 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:14.903491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:14.903925 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:14.903561    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:14.903925 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:14.903608    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:14.903925 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:14.903694    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:16.903187 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:16.903150    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:16.903637 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:16.903150    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:16.903637 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:16.903299    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:16.903637 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:16.903377    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:16.903637 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:16.903158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:16.903637 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:16.903477    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:18.553562 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.553527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.553594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553759    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553774    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553795    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553810    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553838    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:22:34.553803044 +0000 UTC m=+34.137188675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:18.553991 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.553862    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:22:34.553844007 +0000 UTC m=+34.137229642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:18.653898 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.653868    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:18.654088 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.654067    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:18.654168 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.654161    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:22:34.654126099 +0000 UTC m=+34.237511733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:18.903435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.903338    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:18.903435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.903372    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:18.903646 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:18.903355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:18.903646 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.903479    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:18.903646 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.903566    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:18.906993 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:18.903641    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:20.904054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:20.904023    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:20.904516 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:20.904151    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:20.904516 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:20.904209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:20.904516 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:20.904324    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:20.904516 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:20.904381    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:20.904516 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:20.904462    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:22.904065 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:22.903740    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:22.904842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:22.903740    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:22.904842 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:22.904117    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:22.904842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:22.903797    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:22.904842 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:22.904191    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:22.904842 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:22.904240    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:23.126361 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.126327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lflsh" event={"ID":"6d5ac4a2-e29f-4c7f-b9db-d8c021a09487","Type":"ContainerStarted","Data":"f17d1c96fe0c27ca1b7f1049a21c63f504a52d87cd10ec525e5459da74bea07f"}
Sep 18 06:22:23.127769 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.127713    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="c815eb1c0dbdd23a483a4a8ed9209bc74b6a971c678482ef6c2d9f163ae84bb6" exitCode=0
Sep 18 06:22:23.127881 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.127796    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"c815eb1c0dbdd23a483a4a8ed9209bc74b6a971c678482ef6c2d9f163ae84bb6"}
Sep 18 06:22:23.131046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131027    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"d98c57ae0aece7486ab934a933aed96e566a56c39d27486086a1b5e9c270b80c"}
Sep 18 06:22:23.131147 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"4dce0ec4d53e1b4d67a0accb629fb28b01f2468d4c4311f7f00872a9e88cf1ba"}
Sep 18 06:22:23.131147 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"8dfdee14a4f6ce6d24f7e7a519a5c794d3fcf84bb0837adb86347caee1b84357"}
Sep 18 06:22:23.131237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"12d2fc1432e4fc7a5288959ba5a05a32397fb625c2b8a176e8484f19cb893f54"}
Sep 18 06:22:23.131237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131196    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"1d2301ff9cc5c81acf60934c480d3dedd29eefd73f5cfd09e51d09e837eb18c8"}
Sep 18 06:22:23.131237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.131223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"881b2de753e7b83539a3f0afaf3b50530ef4dc8b07bc61c7c44c88d9e02adc4c"}
Sep 18 06:22:23.132428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.132407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69qzh" event={"ID":"2bfbb8e3-a8aa-44f2-9e50-82aac7402a0b","Type":"ContainerStarted","Data":"6c6f133ee91b2093adaae57f8778f67f8a7a0d61e4ca63533756cf52e837882a"}
Sep 18 06:22:23.133880 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.133860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fqclk" event={"ID":"f7f1513d-e978-4051-9a08-56ad33a5e160","Type":"ContainerStarted","Data":"39323f19290d5369688bb0ef818fa193bd8bf90b74f9a7700e7678b1ba0ea617"}
Sep 18 06:22:23.135183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.135160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j" event={"ID":"d5ad6271-e63e-4598-a728-7d268afb74c9","Type":"ContainerStarted","Data":"94dee1d538d0e6303ee4798b748dfb4e234a7652303e9bcb7c1fc890661e6f1d"}
Sep 18 06:22:23.136467 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.136447    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbqkl" event={"ID":"0959cf8e-4192-40bb-9351-5753f6d0e440","Type":"ContainerStarted","Data":"c60ed86760e7233735c2c2f2a87d1df925cb45c955151d553bd94d7daf70b60e"}
Sep 18 06:22:23.137840 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.137821    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jbvbm" event={"ID":"a3644ffb-4d40-476d-ace4-5af038423533","Type":"ContainerStarted","Data":"c7e0e1181311366351722fdaa9c14280bdf08d9333eb21caf72d06789f674d3b"}
Sep 18 06:22:23.139890 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.139853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" podStartSLOduration=21.139842785 podStartE2EDuration="21.139842785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:06.108291914 +0000 UTC m=+5.691677573" watchObservedRunningTime="2026-09-18 06:22:23.139842785 +0000 UTC m=+22.723228437"
Sep 18 06:22:23.156126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.156049    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lflsh" podStartSLOduration=4.184968612 podStartE2EDuration="22.156038422s" totalImagesPullingTime="17.97106981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.139553833 +0000 UTC m=+22.722939485" watchObservedRunningTime="2026-09-18 06:22:23.156038422 +0000 UTC m=+22.739424075"
Sep 18 06:22:23.169126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.169094    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbqkl" podStartSLOduration=4.209292306 podStartE2EDuration="22.169082722s" totalImagesPullingTime="17.959790416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.168682992 +0000 UTC m=+22.752068646" watchObservedRunningTime="2026-09-18 06:22:23.169082722 +0000 UTC m=+22.752468375"
Sep 18 06:22:23.169303 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.169284    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jbvbm" podStartSLOduration=4.143884572 podStartE2EDuration="22.169278785s" totalImagesPullingTime="18.025394213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.155825544 +0000 UTC m=+22.739211196" watchObservedRunningTime="2026-09-18 06:22:23.169278785 +0000 UTC m=+22.752664476"
Sep 18 06:22:23.205332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.205282    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-69qzh" podStartSLOduration=4.21575361 podStartE2EDuration="22.205266895s" totalImagesPullingTime="17.989513285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.204755006 +0000 UTC m=+22.788140659" watchObservedRunningTime="2026-09-18 06:22:23.205266895 +0000 UTC m=+22.788652547"
Sep 18 06:22:23.221169 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.221110    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fqclk" podStartSLOduration=4.243111518 podStartE2EDuration="22.221093512s" totalImagesPullingTime="17.977981994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:23.220548849 +0000 UTC m=+22.803934517" watchObservedRunningTime="2026-09-18 06:22:23.221093512 +0000 UTC m=+22.804479165"
Sep 18 06:22:23.400237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.400206    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:23.400899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.400886    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:23.809383 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.809340    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 06:22:23.866683 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.866571    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T06:22:23.809361446Z","UUID":"5c299435-54a7-423c-afec-5b29aae12167","Handler":null,"Name":"","Endpoint":""}
Sep 18 06:22:23.867959 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.867934    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 06:22:23.868059 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:23.867970    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 06:22:24.142603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.142519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vlxrj" event={"ID":"cad7595d-601f-47c1-9ec6-758e058999d1","Type":"ContainerStarted","Data":"2ecf64f7630659c3231ebd3549c528fdf113ac5f61667fd09aa774b51402d83c"}
Sep 18 06:22:24.144622 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.144593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j" event={"ID":"d5ad6271-e63e-4598-a728-7d268afb74c9","Type":"ContainerStarted","Data":"50aea649e0dd59f112e8246c1e155c504dc3f1759d626f8393ebd21dfbb04b38"}
Sep 18 06:22:24.145271 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.145250    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:24.145419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.145400    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbqkl"
Sep 18 06:22:24.168997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.168953    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vlxrj" podStartSLOduration=5.206571564 podStartE2EDuration="23.168935873s" totalImagesPullingTime="17.962364309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:24.155787363 +0000 UTC m=+23.739173015" watchObservedRunningTime="2026-09-18 06:22:24.168935873 +0000 UTC m=+23.752321527"
Sep 18 06:22:24.903598 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.903566    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:24.903765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.903607    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:24.903765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:24.903681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:24.903765 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:24.903691    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:24.903941 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:24.903795    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:24.903941 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:24.903891    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:25.152276 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:25.152168    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"bb5d532a032a5db4a9536932cf3317870e69e7c36e315194a4d163dbef1075ce"}
Sep 18 06:22:26.157457 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:26.157415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j" event={"ID":"d5ad6271-e63e-4598-a728-7d268afb74c9","Type":"ContainerStarted","Data":"41c030f42020a103e1f35e8a9a263963edb0e0731aec2c9d9b3bd69e8bf0b5cf"}
Sep 18 06:22:26.187290 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:26.187242    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qns5j" podStartSLOduration=4.209983663 podStartE2EDuration="25.187226281s" totalImagesPullingTime="20.977242618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:26.184850524 +0000 UTC m=+25.768236178" watchObservedRunningTime="2026-09-18 06:22:26.187226281 +0000 UTC m=+25.770611934"
Sep 18 06:22:26.907344 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:26.907313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:26.907344 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:26.907345    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:26.907575 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:26.907435    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:26.907746 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:26.907726    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:26.907854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:26.907832    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:26.907966 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:26.907888    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:28.164393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.164118    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="7050bc1d98c358cf3aff76de16912c10dfd0b24ab106b293d71f40d985efa562" exitCode=0
Sep 18 06:22:28.165252 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.164180    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"7050bc1d98c358cf3aff76de16912c10dfd0b24ab106b293d71f40d985efa562"}
Sep 18 06:22:28.168453 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.168424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" event={"ID":"28597235-7299-4269-8e10-45bb68e87f84","Type":"ContainerStarted","Data":"ea0cbdb3af8114b41295b4b054296b00ad4eacb3c5d5c09f3a9b8677cfffa9f2"}
Sep 18 06:22:28.168657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.168640    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:28.168745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.168716    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:28.168801 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.168753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:28.183308 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.183287    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:28.183545 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.183529    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:22:28.207585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.207541    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p645l" podStartSLOduration=9.186121421 podStartE2EDuration="27.207526635s" totalImagesPullingTime="18.021405214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:28.207158333 +0000 UTC m=+27.790543986" watchObservedRunningTime="2026-09-18 06:22:28.207526635 +0000 UTC m=+27.790912289"
Sep 18 06:22:28.903845 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.903638    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:28.903944 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.903638    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:28.904002 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:28.903953    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:28.904052 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:28.904008    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:28.904052 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:28.903637    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:28.904184 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:28.904074    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:29.180322 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.180103    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="1f74635ec3c415cd5ed1d407db862b72407ed1ca49900f45daa407f9bf569cc4" exitCode=0
Sep 18 06:22:29.180322 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.180171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"1f74635ec3c415cd5ed1d407db862b72407ed1ca49900f45daa407f9bf569cc4"}
Sep 18 06:22:29.847365 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.846673    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k6qw9"]
Sep 18 06:22:29.847365 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.846801    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:29.847365 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:29.846932    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.849850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rk2dg"]
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.849937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwqdd"]
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.850042    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:29.850193    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:29.850258    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:29.852045 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:29.850350    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:30.184891 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:30.184820    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="be58843af2aef6107e271dfe1a0716000100eaef733fac6d389890d327a1013c" exitCode=0
Sep 18 06:22:30.185245 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:30.184906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"be58843af2aef6107e271dfe1a0716000100eaef733fac6d389890d327a1013c"}
Sep 18 06:22:31.903525 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:31.903447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:31.903939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:31.903458    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:31.903939 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:31.903592    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:31.903939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:31.903631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:31.903939 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:31.903731    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:31.903939 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:31.903803    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:33.903220 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:33.903182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:33.903220 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:33.903215    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:33.903716 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:33.903226    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:33.903716 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:33.903300    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k6qw9" podUID="415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f"
Sep 18 06:22:33.903716 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:33.903400    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rk2dg" podUID="50db7a18-9223-4a59-b168-f07f6e3fa38a"
Sep 18 06:22:33.903716 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:33.903468    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:22:34.572220 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:34.572153    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:34.572243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572265    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572291    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572304    2567 projected.go:196] Error preparing data for projected volume kube-api-access-lm28n for pod openshift-network-diagnostics/network-check-target-rk2dg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572344    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572368    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n podName:50db7a18-9223-4a59-b168-f07f6e3fa38a nodeName:}" failed. No retries permitted until 2026-09-18 06:23:06.572351348 +0000 UTC m=+66.155736999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lm28n" (UniqueName: "kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n") pod "network-check-target-rk2dg" (UID: "50db7a18-9223-4a59-b168-f07f6e3fa38a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 06:22:34.572446 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.572390    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:23:06.572380333 +0000 UTC m=+66.155765964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 06:22:34.672822 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:34.672793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:34.672952 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.672912    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:34.673003 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:34.672969    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret podName:415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f nodeName:}" failed. No retries permitted until 2026-09-18 06:23:06.672954199 +0000 UTC m=+66.256339843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret") pod "global-pull-secret-syncer-k6qw9" (UID: "415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f") : object "kube-system"/"original-pull-secret" not registered
Sep 18 06:22:35.240149 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.239978    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeReady"
Sep 18 06:22:35.240575 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.240275    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 06:22:35.274346 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.274265    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"]
Sep 18 06:22:35.307537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.307509    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:22:35.307695 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.307613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.310041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.310011    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 06:22:35.310287 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.310272    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6bzkx\""
Sep 18 06:22:35.310386 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.310315    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 06:22:35.321759 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.321738    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"]
Sep 18 06:22:35.321862 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.321767    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7cjhw"]
Sep 18 06:22:35.321862 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.321847    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.324354 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.324337    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 06:22:35.324569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.324548    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 06:22:35.324638 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.324554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7llw\""
Sep 18 06:22:35.324638 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.324554    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 06:22:35.330330 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.330313    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 06:22:35.351836 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.351811    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:22:35.351924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.351849    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vhs6h"]
Sep 18 06:22:35.351924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.351891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.354469 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.354442    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 06:22:35.354564 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.354504    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sn4gt\""
Sep 18 06:22:35.354564 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.354442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 06:22:35.372013 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.371993    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhs6h"]
Sep 18 06:22:35.372099 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.372023    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cjhw"]
Sep 18 06:22:35.372152 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.372105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.376016 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.375999    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 06:22:35.376016 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.376007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2l9zb\""
Sep 18 06:22:35.376166 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.376005    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 06:22:35.376166 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.376104    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 06:22:35.479952 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.479924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.479966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9343c376-e916-43cb-97fd-3002665a86d3-cert\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.479985    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7kk\" (UniqueName: \"kubernetes.io/projected/9343c376-e916-43cb-97fd-3002665a86d3-kube-api-access-5x7kk\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480002    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480055    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsq6\" (UniqueName: \"kubernetes.io/projected/29bfe3a4-5b99-4e12-8574-e2bc18699d64-kube-api-access-zjsq6\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480096    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480115    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480131    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480199    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bfe3a4-5b99-4e12-8574-e2bc18699d64-tmp-dir\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8c6630-d359-48a6-b8c4-057493ecd834-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtw7\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfe3a4-5b99-4e12-8574-e2bc18699d64-config-volume\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.480388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480368    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.480644 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.480400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.581419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581333    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfe3a4-5b99-4e12-8574-e2bc18699d64-config-volume\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.581419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581484    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.581496    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581536    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.581566    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:36.081548166 +0000 UTC m=+35.664933811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9343c376-e916-43cb-97fd-3002665a86d3-cert\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.581628 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.581624    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.581685    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:36.08166952 +0000 UTC m=+35.665055166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581624    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7kk\" (UniqueName: \"kubernetes.io/projected/9343c376-e916-43cb-97fd-3002665a86d3-kube-api-access-5x7kk\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581737    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsq6\" (UniqueName: \"kubernetes.io/projected/29bfe3a4-5b99-4e12-8574-e2bc18699d64-kube-api-access-zjsq6\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581836    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581910    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bfe3a4-5b99-4e12-8574-e2bc18699d64-tmp-dir\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.581951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8c6630-d359-48a6-b8c4-057493ecd834-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.581964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtw7\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.581993    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.582006    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:35.582044    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:36.082029894 +0000 UTC m=+35.665415541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.582075    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/29bfe3a4-5b99-4e12-8574-e2bc18699d64-config-volume\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.582335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bfe3a4-5b99-4e12-8574-e2bc18699d64-tmp-dir\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.582477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.582419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.582784 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.582638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8c6630-d359-48a6-b8c4-057493ecd834-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:35.583072 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.583043    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.585743 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.585722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.585857 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.585819    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.585985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.585966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9343c376-e916-43cb-97fd-3002665a86d3-cert\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.590045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.590026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.590228 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.590199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7kk\" (UniqueName: \"kubernetes.io/projected/9343c376-e916-43cb-97fd-3002665a86d3-kube-api-access-5x7kk\") pod \"ingress-canary-vhs6h\" (UID: \"9343c376-e916-43cb-97fd-3002665a86d3\") " pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.590552 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.590535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsq6\" (UniqueName: \"kubernetes.io/projected/29bfe3a4-5b99-4e12-8574-e2bc18699d64-kube-api-access-zjsq6\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:35.590687 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.590668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtw7\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:35.680444 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.680400    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vhs6h"
Sep 18 06:22:35.903312 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.903284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:22:35.903449 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.903328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:22:35.903449 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.903404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:22:35.906303 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906284    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fg7cb\""
Sep 18 06:22:35.906409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906363    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kj54z\""
Sep 18 06:22:35.906409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906363    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 06:22:35.906409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906401    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 06:22:35.906547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906448    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 06:22:35.906547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:35.906508    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 06:22:36.015361 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.015332    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vhs6h"]
Sep 18 06:22:36.019471 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:22:36.019445    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9343c376_e916_43cb_97fd_3002665a86d3.slice/crio-6a9e3c5946659ee16d0a8e949793fbd1ae433bcb203c301ea398a5a6b9cfcf0e WatchSource:0}: Error finding container 6a9e3c5946659ee16d0a8e949793fbd1ae433bcb203c301ea398a5a6b9cfcf0e: Status 404 returned error can't find the container with id 6a9e3c5946659ee16d0a8e949793fbd1ae433bcb203c301ea398a5a6b9cfcf0e
Sep 18 06:22:36.086255 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.086190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:36.086255 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.086231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:36.086392 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.086261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:36.086392 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086369    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:36.086392 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086370    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:36.086496 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086374    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:36.086496 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086419    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:37.086405675 +0000 UTC m=+36.669791305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:36.086496 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086421    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:36.086496 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086432    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:37.086426435 +0000 UTC m=+36.669812066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:36.086496 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:36.086458    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:37.086441212 +0000 UTC m=+36.669826858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:22:36.201588 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.201558    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhs6h" event={"ID":"9343c376-e916-43cb-97fd-3002665a86d3","Type":"ContainerStarted","Data":"6a9e3c5946659ee16d0a8e949793fbd1ae433bcb203c301ea398a5a6b9cfcf0e"}
Sep 18 06:22:36.207346 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.207317    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="c7b5d3c36765a904cb82faf57d7fa15854cb7507fce7de7d451da6e8ee180cb5" exitCode=0
Sep 18 06:22:36.207461 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:36.207354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"c7b5d3c36765a904cb82faf57d7fa15854cb7507fce7de7d451da6e8ee180cb5"}
Sep 18 06:22:37.094044 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:37.093828    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:37.094076    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.093990    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:37.094117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094126    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094207    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:39.094185917 +0000 UTC m=+38.677571563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094226    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094268    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094302    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:39.09428481 +0000 UTC m=+38.677670444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:37.094854 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:37.094324    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:39.094313689 +0000 UTC m=+38.677699326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:37.213539 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:37.213501    2567 generic.go:356] "Generic (PLEG): container finished" podID="b395dace-429c-4e52-9e3e-ca1328cc8320" containerID="67e01edafdf8e143961e7938ca81aad5b87144734f446071f7abee7f3ea7eba2" exitCode=0
Sep 18 06:22:37.213721 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:37.213571    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerDied","Data":"67e01edafdf8e143961e7938ca81aad5b87144734f446071f7abee7f3ea7eba2"}
Sep 18 06:22:38.220025 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:38.219956    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n75xx" event={"ID":"b395dace-429c-4e52-9e3e-ca1328cc8320","Type":"ContainerStarted","Data":"1c93e3b743e983f8c91bb63da5a75e18811436ef8f9be132439b1b6355bcbe5a"}
Sep 18 06:22:38.241751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:38.241575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n75xx" podStartSLOduration=7.524712231 podStartE2EDuration="37.241555973s" totalImagesPullingTime="29.716843742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:38.239576181 +0000 UTC m=+37.822961835" watchObservedRunningTime="2026-09-18 06:22:38.241555973 +0000 UTC m=+37.824941630"
Sep 18 06:22:39.113413 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:39.113371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:39.113587 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:39.113422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:39.113587 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:39.113476    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:39.113587 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113541    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:39.113587 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113562    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:39.113587 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113573    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:39.113782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113600    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:39.113782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113615    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:43.113597402 +0000 UTC m=+42.696983053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:39.113782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113632    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:43.113625848 +0000 UTC m=+42.697011479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:22:39.113782 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:39.113655    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:43.113637647 +0000 UTC m=+42.697023284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:39.224069 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:39.224033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vhs6h" event={"ID":"9343c376-e916-43cb-97fd-3002665a86d3","Type":"ContainerStarted","Data":"bb9a08b3e2c7a31a63e6ac8c175b536582e4d1ab8abbdffa4511e8b65d1cdd31"}
Sep 18 06:22:39.238150 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:39.238099    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vhs6h" podStartSLOduration=1.5594440550000002 podStartE2EDuration="4.238085137s" totalImagesPullingTime="2.678641082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:22:39.237432311 +0000 UTC m=+38.820817964" watchObservedRunningTime="2026-09-18 06:22:39.238085137 +0000 UTC m=+38.821470780"
Sep 18 06:22:43.145732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:43.145690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:43.145732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:43.145738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:43.145790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145834    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145877    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145889    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145911    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:51.145894096 +0000 UTC m=+50.729279727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145929    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:51.14591816 +0000 UTC m=+50.729303790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145877    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:43.146160 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:43.145956    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:22:51.145947797 +0000 UTC m=+50.729333428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:51.192638 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:51.192605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:51.192646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:22:51.192674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192760    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192768    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192816    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:07.192802464 +0000 UTC m=+66.776188096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192842    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:07.192826238 +0000 UTC m=+66.776211870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192771    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192857    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:22:51.193079 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:22:51.192892    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:07.192882118 +0000 UTC m=+66.776267754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:23:00.195445 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:00.195416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p645l"
Sep 18 06:23:06.598551 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.598501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:23:06.598949 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.598568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:23:06.601415 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.601394    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 06:23:06.601659 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.601645    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 06:23:06.609700 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:06.609679    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 06:23:06.609760 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:06.609752    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:24:10.60973022 +0000 UTC m=+130.193115855 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : secret "metrics-daemon-secret" not found
Sep 18 06:23:06.611889 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.611870    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 06:23:06.623998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.623971    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm28n\" (UniqueName: \"kubernetes.io/projected/50db7a18-9223-4a59-b168-f07f6e3fa38a-kube-api-access-lm28n\") pod \"network-check-target-rk2dg\" (UID: \"50db7a18-9223-4a59-b168-f07f6e3fa38a\") " pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:23:06.699269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.699229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:23:06.701949 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.701928    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 06:23:06.712045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.712022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f-original-pull-secret\") pod \"global-pull-secret-syncer-k6qw9\" (UID: \"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f\") " pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:23:06.818870 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.818819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k6qw9"
Sep 18 06:23:06.826420 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.826402    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kj54z\""
Sep 18 06:23:06.834098 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:06.834078    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:23:07.089419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.089391    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k6qw9"]
Sep 18 06:23:07.091971 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:23:07.091938    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415bbda1_0b2a_45c3_8ae1_6d10f8e7da8f.slice/crio-45f2a64e242c98f15eb09f84d31cc20039fb67e722daaabd6989d21cbb8456a4 WatchSource:0}: Error finding container 45f2a64e242c98f15eb09f84d31cc20039fb67e722daaabd6989d21cbb8456a4: Status 404 returned error can't find the container with id 45f2a64e242c98f15eb09f84d31cc20039fb67e722daaabd6989d21cbb8456a4
Sep 18 06:23:07.105597 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.105578    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rk2dg"]
Sep 18 06:23:07.108643 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:23:07.108622    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50db7a18_9223_4a59_b168_f07f6e3fa38a.slice/crio-b5dd5fcb333f996b8821bbbb4a108352e3cd1558fe8835b0ae006b1c56fa09b8 WatchSource:0}: Error finding container b5dd5fcb333f996b8821bbbb4a108352e3cd1558fe8835b0ae006b1c56fa09b8: Status 404 returned error can't find the container with id b5dd5fcb333f996b8821bbbb4a108352e3cd1558fe8835b0ae006b1c56fa09b8
Sep 18 06:23:07.203426 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.203404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:23:07.203571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.203444    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:23:07.203571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.203482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:23:07.203571 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203553    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:23:07.203683 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203572    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:23:07.203683 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203577    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:23:07.203683 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203608    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:39.203592365 +0000 UTC m=+98.786977996 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:23:07.203683 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203642    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:39.203628807 +0000 UTC m=+98.787014443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:23:07.203683 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203582    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:23:07.203865 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:07.203696    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:23:39.203686355 +0000 UTC m=+98.787071986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:23:07.289589 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.289549    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k6qw9" event={"ID":"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f","Type":"ContainerStarted","Data":"45f2a64e242c98f15eb09f84d31cc20039fb67e722daaabd6989d21cbb8456a4"}
Sep 18 06:23:07.290772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:07.290750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rk2dg" event={"ID":"50db7a18-9223-4a59-b168-f07f6e3fa38a","Type":"ContainerStarted","Data":"b5dd5fcb333f996b8821bbbb4a108352e3cd1558fe8835b0ae006b1c56fa09b8"}
Sep 18 06:23:12.305854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:12.305815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rk2dg" event={"ID":"50db7a18-9223-4a59-b168-f07f6e3fa38a","Type":"ContainerStarted","Data":"d74984281c43c48aca157334132ebb39898f3ee609d94ce670c90fd9cfc8eba8"}
Sep 18 06:23:12.306421 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:12.305931    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:23:12.307256 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:12.307234    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k6qw9" event={"ID":"415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f","Type":"ContainerStarted","Data":"d3bc7ff630654b9c15ce21be9049f1c53fd02e21a1151aa5672374d43ac6171b"}
Sep 18 06:23:12.320521 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:12.320481    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rk2dg" podStartSLOduration=66.504299802 podStartE2EDuration="1m11.320469674s" totalImagesPullingTime="4.816169872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:12.319668136 +0000 UTC m=+71.903053791" watchObservedRunningTime="2026-09-18 06:23:12.320469674 +0000 UTC m=+71.903855364"
Sep 18 06:23:12.333959 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:12.333916    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k6qw9" podStartSLOduration=66.497768653 podStartE2EDuration="1m11.333902634s" totalImagesPullingTime="4.836133981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:12.333262956 +0000 UTC m=+71.916648621" watchObservedRunningTime="2026-09-18 06:23:12.333902634 +0000 UTC m=+71.917288286"
Sep 18 06:23:33.926011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.925900    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"]
Sep 18 06:23:33.928400 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.928382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:33.930840 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.930818    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 06:23:33.930966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.930844    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 06:23:33.932046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.932030    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 06:23:33.932191 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.932031    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 06:23:33.939199 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.939180    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"]
Sep 18 06:23:33.986947 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.986908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc89360c-983f-4284-818b-291d53d9e5ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:33.986947 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.986949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvpt\" (UniqueName: \"kubernetes.io/projected/bc89360c-983f-4284-818b-291d53d9e5ec-kube-api-access-2gvpt\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:33.987121 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:33.987035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc89360c-983f-4284-818b-291d53d9e5ec-tmp\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.087811 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.087766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc89360c-983f-4284-818b-291d53d9e5ec-tmp\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.087912 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.087834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc89360c-983f-4284-818b-291d53d9e5ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.087912 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.087853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvpt\" (UniqueName: \"kubernetes.io/projected/bc89360c-983f-4284-818b-291d53d9e5ec-kube-api-access-2gvpt\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.088324 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.088297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc89360c-983f-4284-818b-291d53d9e5ec-tmp\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.090293 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.090268    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bc89360c-983f-4284-818b-291d53d9e5ec-klusterlet-config\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.094850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.094828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvpt\" (UniqueName: \"kubernetes.io/projected/bc89360c-983f-4284-818b-291d53d9e5ec-kube-api-access-2gvpt\") pod \"klusterlet-addon-workmgr-59f65744fc-vf6z4\" (UID: \"bc89360c-983f-4284-818b-291d53d9e5ec\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.237526 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.237491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:34.502801 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:34.502734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"]
Sep 18 06:23:34.506208 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:23:34.506172    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc89360c_983f_4284_818b_291d53d9e5ec.slice/crio-70e12a1887cc75833299c69da41c9053d07c859c4ee08ad2eab799642b9472e1 WatchSource:0}: Error finding container 70e12a1887cc75833299c69da41c9053d07c859c4ee08ad2eab799642b9472e1: Status 404 returned error can't find the container with id 70e12a1887cc75833299c69da41c9053d07c859c4ee08ad2eab799642b9472e1
Sep 18 06:23:35.364592 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:35.364553    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" event={"ID":"bc89360c-983f-4284-818b-291d53d9e5ec","Type":"ContainerStarted","Data":"70e12a1887cc75833299c69da41c9053d07c859c4ee08ad2eab799642b9472e1"}
Sep 18 06:23:38.374160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:38.374063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" event={"ID":"bc89360c-983f-4284-818b-291d53d9e5ec","Type":"ContainerStarted","Data":"3e1850450ad325128f91107884346c6483cee2bf5cda487ae40481afc9036ba3"}
Sep 18 06:23:38.374501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:38.374302    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:38.375828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:38.375811    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:23:38.390214 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:38.390173    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" podStartSLOduration=1.775257593 podStartE2EDuration="5.390161228s" totalImagesPullingTime="3.614903635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:23:38.388616256 +0000 UTC m=+97.972001908" watchObservedRunningTime="2026-09-18 06:23:38.390161228 +0000 UTC m=+97.973546883"
Sep 18 06:23:39.233349 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:39.233309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:23:39.233349 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:39.233356    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:39.233381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233467    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233473    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233476    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233522    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:24:43.233508959 +0000 UTC m=+162.816894589 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233527    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233534    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:24:43.233529048 +0000 UTC m=+162.816914679 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:23:39.233560 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:23:39.233555    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:24:43.233541797 +0000 UTC m=+162.816927428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:23:43.312242 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:23:43.312208    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rk2dg"
Sep 18 06:24:10.672739 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:10.672703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:24:10.673241 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:10.672819    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 06:24:10.673241 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:10.672882    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:26:12.672867309 +0000 UTC m=+252.256252952 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : secret "metrics-daemon-secret" not found
Sep 18 06:24:38.318453 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:38.318407    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" podUID="8c8c6630-d359-48a6-b8c4-057493ecd834"
Sep 18 06:24:38.329613 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:38.329582    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1"
Sep 18 06:24:38.359848 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:38.359824    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7cjhw" podUID="29bfe3a4-5b99-4e12-8574-e2bc18699d64"
Sep 18 06:24:38.525382 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:38.525352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:24:38.525557 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:38.525355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:24:38.525557 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:38.525355    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:24:38.913298 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:38.913266    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:24:43.295786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:43.295746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:43.295798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:43.295851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.295941    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.295940    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.295977    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.295953    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68d84f4889-wggh4: secret "image-registry-tls" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.296040    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls podName:29bfe3a4-5b99-4e12-8574-e2bc18699d64 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:45.296019182 +0000 UTC m=+284.879404820 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls") pod "dns-default-7cjhw" (UID: "29bfe3a4-5b99-4e12-8574-e2bc18699d64") : secret "dns-default-metrics-tls" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.296058    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:45.296049402 +0000 UTC m=+284.879435034 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:24:43.296194 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:24:43.296068    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls podName:cb144779-35ac-443b-8c3b-0fa81bf95ca1 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:45.296062676 +0000 UTC m=+284.879448306 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls") pod "image-registry-68d84f4889-wggh4" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1") : secret "image-registry-tls" not found
Sep 18 06:24:51.903729 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:24:51.903684    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:25:38.375100 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:38.375051    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" podUID="bc89360c-983f-4284-818b-291d53d9e5ec" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 18 06:25:38.671929 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:38.671844    2567 generic.go:356] "Generic (PLEG): container finished" podID="bc89360c-983f-4284-818b-291d53d9e5ec" containerID="3e1850450ad325128f91107884346c6483cee2bf5cda487ae40481afc9036ba3" exitCode=1
Sep 18 06:25:38.671929 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:38.671881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" event={"ID":"bc89360c-983f-4284-818b-291d53d9e5ec","Type":"ContainerDied","Data":"3e1850450ad325128f91107884346c6483cee2bf5cda487ae40481afc9036ba3"}
Sep 18 06:25:38.672206 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:38.672184    2567 scope.go:117] "RemoveContainer" containerID="3e1850450ad325128f91107884346c6483cee2bf5cda487ae40481afc9036ba3"
Sep 18 06:25:39.675674 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:39.675642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4" event={"ID":"bc89360c-983f-4284-818b-291d53d9e5ec","Type":"ContainerStarted","Data":"bfc71dc8a48185b5288fefc589fc9433b12a11f9045ac434b71b0728223b028c"}
Sep 18 06:25:39.676088 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:39.675929    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:25:39.676517 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:25:39.676498    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59f65744fc-vf6z4"
Sep 18 06:26:12.707775 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:12.707730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:26:12.708244 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:12.707888    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 06:26:12.708244 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:12.707966    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs podName:7f327502-c45b-4c98-802a-a4ad54e6fbfa nodeName:}" failed. No retries permitted until 2026-09-18 06:28:14.707948422 +0000 UTC m=+374.291334052 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs") pod "network-metrics-daemon-fwqdd" (UID: "7f327502-c45b-4c98-802a-a4ad54e6fbfa") : secret "metrics-daemon-secret" not found
Sep 18 06:26:26.910548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.910474    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"]
Sep 18 06:26:26.913346 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.913329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:26.915897 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.915875    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 06:26:26.916002 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.915877    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 06:26:26.917016 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.917001    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 06:26:26.917117 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.917030    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 06:26:26.918130 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.918108    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gcbjb\""
Sep 18 06:26:26.923284 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:26.923254    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"]
Sep 18 06:26:27.008621 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.008590    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kggmd"]
Sep 18 06:26:27.011418 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.011403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.013976 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.013952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lj7pq\""
Sep 18 06:26:27.014097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.013953    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 06:26:27.014097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.013953    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 06:26:27.014296 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.014103    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 06:26:27.014632 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.014619    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 06:26:27.020555 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.020531    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 06:26:27.021524 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.021505    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kggmd"]
Sep 18 06:26:27.100195 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100161    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22febb06-4e87-4056-aa22-bd068da7eb94-config\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.100195 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4327d4f3-0131-4777-b6d5-df915eab4b4e-serving-cert\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.100404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrll\" (UniqueName: \"kubernetes.io/projected/4327d4f3-0131-4777-b6d5-df915eab4b4e-kube-api-access-xtrll\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.100404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx59q\" (UniqueName: \"kubernetes.io/projected/22febb06-4e87-4056-aa22-bd068da7eb94-kube-api-access-vx59q\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.100404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100265    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-trusted-ca\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.100404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22febb06-4e87-4056-aa22-bd068da7eb94-serving-cert\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.100404 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.100327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-config\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.201233 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22febb06-4e87-4056-aa22-bd068da7eb94-config\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.201339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4327d4f3-0131-4777-b6d5-df915eab4b4e-serving-cert\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.201339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrll\" (UniqueName: \"kubernetes.io/projected/4327d4f3-0131-4777-b6d5-df915eab4b4e-kube-api-access-xtrll\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.201429 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx59q\" (UniqueName: \"kubernetes.io/projected/22febb06-4e87-4056-aa22-bd068da7eb94-kube-api-access-vx59q\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.201476 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-trusted-ca\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.201542 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22febb06-4e87-4056-aa22-bd068da7eb94-serving-cert\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.201589 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-config\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.201903 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.201871    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22febb06-4e87-4056-aa22-bd068da7eb94-config\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.202313 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.202294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-trusted-ca\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.202386 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.202320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4327d4f3-0131-4777-b6d5-df915eab4b4e-config\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.203714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.203691    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22febb06-4e87-4056-aa22-bd068da7eb94-serving-cert\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.203806 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.203703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4327d4f3-0131-4777-b6d5-df915eab4b4e-serving-cert\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.208483 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.208457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrll\" (UniqueName: \"kubernetes.io/projected/4327d4f3-0131-4777-b6d5-df915eab4b4e-kube-api-access-xtrll\") pod \"console-operator-fd4bb46c-kggmd\" (UID: \"4327d4f3-0131-4777-b6d5-df915eab4b4e\") " pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.208582 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.208463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx59q\" (UniqueName: \"kubernetes.io/projected/22febb06-4e87-4056-aa22-bd068da7eb94-kube-api-access-vx59q\") pod \"service-ca-operator-57cdc4b768-fhg9k\" (UID: \"22febb06-4e87-4056-aa22-bd068da7eb94\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.221351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.221330    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"
Sep 18 06:26:27.319798 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.319754    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:27.487948 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.487872    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k"]
Sep 18 06:26:27.491203 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:26:27.491177    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22febb06_4e87_4056_aa22_bd068da7eb94.slice/crio-1d0a8ef3d14de76fce1002ab6bb1932b2a2ec264e2bdfdfdb6bfc2644c09c687 WatchSource:0}: Error finding container 1d0a8ef3d14de76fce1002ab6bb1932b2a2ec264e2bdfdfdb6bfc2644c09c687: Status 404 returned error can't find the container with id 1d0a8ef3d14de76fce1002ab6bb1932b2a2ec264e2bdfdfdb6bfc2644c09c687
Sep 18 06:26:27.586580 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.586548    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-kggmd"]
Sep 18 06:26:27.589410 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:26:27.589385    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4327d4f3_0131_4777_b6d5_df915eab4b4e.slice/crio-393dd68a9a0cb5ea4fea45995d010f46af93222f222c18e37aef0773a47e81d5 WatchSource:0}: Error finding container 393dd68a9a0cb5ea4fea45995d010f46af93222f222c18e37aef0773a47e81d5: Status 404 returned error can't find the container with id 393dd68a9a0cb5ea4fea45995d010f46af93222f222c18e37aef0773a47e81d5
Sep 18 06:26:27.794380 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.794292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k" event={"ID":"22febb06-4e87-4056-aa22-bd068da7eb94","Type":"ContainerStarted","Data":"1d0a8ef3d14de76fce1002ab6bb1932b2a2ec264e2bdfdfdb6bfc2644c09c687"}
Sep 18 06:26:27.795294 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:27.795272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" event={"ID":"4327d4f3-0131-4777-b6d5-df915eab4b4e","Type":"ContainerStarted","Data":"393dd68a9a0cb5ea4fea45995d010f46af93222f222c18e37aef0773a47e81d5"}
Sep 18 06:26:30.806602 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.806566    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k" event={"ID":"22febb06-4e87-4056-aa22-bd068da7eb94","Type":"ContainerStarted","Data":"2df0d9fc76566ce240d001cc697820f98c1cfa5cf06a7581a68893817b8ac56d"}
Sep 18 06:26:30.808293 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.808271    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/0.log"
Sep 18 06:26:30.808419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.808312    2567 generic.go:356] "Generic (PLEG): container finished" podID="4327d4f3-0131-4777-b6d5-df915eab4b4e" containerID="50144e6c313908793ab13cfc83f203ffa3ae5c091e50fb082c6b142e71f93e6a" exitCode=255
Sep 18 06:26:30.808419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.808348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" event={"ID":"4327d4f3-0131-4777-b6d5-df915eab4b4e","Type":"ContainerDied","Data":"50144e6c313908793ab13cfc83f203ffa3ae5c091e50fb082c6b142e71f93e6a"}
Sep 18 06:26:30.808590 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.808576    2567 scope.go:117] "RemoveContainer" containerID="50144e6c313908793ab13cfc83f203ffa3ae5c091e50fb082c6b142e71f93e6a"
Sep 18 06:26:30.822196 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:30.822108    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k" podStartSLOduration=2.353432048 podStartE2EDuration="4.822093201s" totalImagesPullingTime="2.468661153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:30.821111838 +0000 UTC m=+270.404497495" watchObservedRunningTime="2026-09-18 06:26:30.822093201 +0000 UTC m=+270.405478854"
Sep 18 06:26:31.503524 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.503488    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"]
Sep 18 06:26:31.506510 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.506492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"
Sep 18 06:26:31.508836 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.508806    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 06:26:31.508947 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.508924    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-d22xr\""
Sep 18 06:26:31.509013 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.508969    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 06:26:31.517439 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.517414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"]
Sep 18 06:26:31.633216 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.633177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6pz\" (UniqueName: \"kubernetes.io/projected/5ac92020-4c0b-4ad2-85b0-aecd0dea69de-kube-api-access-qv6pz\") pod \"migrator-76794cbd6c-jgzjg\" (UID: \"5ac92020-4c0b-4ad2-85b0-aecd0dea69de\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"
Sep 18 06:26:31.734432 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.734393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6pz\" (UniqueName: \"kubernetes.io/projected/5ac92020-4c0b-4ad2-85b0-aecd0dea69de-kube-api-access-qv6pz\") pod \"migrator-76794cbd6c-jgzjg\" (UID: \"5ac92020-4c0b-4ad2-85b0-aecd0dea69de\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"
Sep 18 06:26:31.742376 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.742351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6pz\" (UniqueName: \"kubernetes.io/projected/5ac92020-4c0b-4ad2-85b0-aecd0dea69de-kube-api-access-qv6pz\") pod \"migrator-76794cbd6c-jgzjg\" (UID: \"5ac92020-4c0b-4ad2-85b0-aecd0dea69de\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"
Sep 18 06:26:31.813822 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.813754    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/1.log"
Sep 18 06:26:31.814193 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814165    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/0.log"
Sep 18 06:26:31.814245 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814198    2567 generic.go:356] "Generic (PLEG): container finished" podID="4327d4f3-0131-4777-b6d5-df915eab4b4e" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5" exitCode=255
Sep 18 06:26:31.814303 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814281    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" event={"ID":"4327d4f3-0131-4777-b6d5-df915eab4b4e","Type":"ContainerDied","Data":"125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"}
Sep 18 06:26:31.814344 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814334    2567 scope.go:117] "RemoveContainer" containerID="50144e6c313908793ab13cfc83f203ffa3ae5c091e50fb082c6b142e71f93e6a"
Sep 18 06:26:31.814589 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814574    2567 scope.go:117] "RemoveContainer" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"
Sep 18 06:26:31.814735 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.814715    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"
Sep 18 06:26:31.814851 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:31.814744    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:26:31.944010 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.943979    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-45psb"]
Sep 18 06:26:31.948309 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.948291    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:31.950615 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.950592    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 06:26:31.950713 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.950659    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 06:26:31.950910 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.950882    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 06:26:31.951007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.950927    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 06:26:31.951093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.951080    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6ts\""
Sep 18 06:26:31.958235 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:31.958210    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-45psb"]
Sep 18 06:26:32.080046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.079978    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg"]
Sep 18 06:26:32.082969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:26:32.082940    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac92020_4c0b_4ad2_85b0_aecd0dea69de.slice/crio-ab4905ec91a10f4d29c58b1b3aa4cc446ac3f0822cd6f6f70203eae9600ca6ab WatchSource:0}: Error finding container ab4905ec91a10f4d29c58b1b3aa4cc446ac3f0822cd6f6f70203eae9600ca6ab: Status 404 returned error can't find the container with id ab4905ec91a10f4d29c58b1b3aa4cc446ac3f0822cd6f6f70203eae9600ca6ab
Sep 18 06:26:32.137864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.137835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzkb\" (UniqueName: \"kubernetes.io/projected/9c27008c-d284-4b5e-8575-a368c862ec86-kube-api-access-bpzkb\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.137969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.137890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c27008c-d284-4b5e-8575-a368c862ec86-crio-socket\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.137969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.137935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.137969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.137951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.138125 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.138024    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c27008c-d284-4b5e-8575-a368c862ec86-data-volume\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240612 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240571    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c27008c-d284-4b5e-8575-a368c862ec86-data-volume\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzkb\" (UniqueName: \"kubernetes.io/projected/9c27008c-d284-4b5e-8575-a368c862ec86-kube-api-access-bpzkb\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c27008c-d284-4b5e-8575-a368c862ec86-crio-socket\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.240799    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240838    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c27008c-d284-4b5e-8575-a368c862ec86-crio-socket\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.240867    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:32.740849205 +0000 UTC m=+272.324234858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : configmap "kube-rbac-proxy" not found
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.240915    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.240955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c27008c-d284-4b5e-8575-a368c862ec86-data-volume\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.240980 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.240970    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:32.740954335 +0000 UTC m=+272.324339967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:32.249972 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.249950    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzkb\" (UniqueName: \"kubernetes.io/projected/9c27008c-d284-4b5e-8575-a368c862ec86-kube-api-access-bpzkb\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.745223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.745185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.745223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.745227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:32.745490 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.745466    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 06:26:32.745559 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.745538    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:32.745559 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.745559    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:33.745533085 +0000 UTC m=+273.328918733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : configmap "kube-rbac-proxy" not found
Sep 18 06:26:32.745700 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.745601    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:33.745588015 +0000 UTC m=+273.328973651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:32.818954 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.818925    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/1.log"
Sep 18 06:26:32.819395 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.819290    2567 scope.go:117] "RemoveContainer" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"
Sep 18 06:26:32.819483 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:32.819463    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:26:32.820238 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:32.820213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg" event={"ID":"5ac92020-4c0b-4ad2-85b0-aecd0dea69de","Type":"ContainerStarted","Data":"ab4905ec91a10f4d29c58b1b3aa4cc446ac3f0822cd6f6f70203eae9600ca6ab"}
Sep 18 06:26:33.753703 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:33.753671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:33.753703 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:33.753707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:33.753973 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:33.753818    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 06:26:33.753973 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:33.753846    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:33.753973 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:33.753893    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:35.753874425 +0000 UTC m=+275.337260073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : configmap "kube-rbac-proxy" not found
Sep 18 06:26:33.753973 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:33.753913    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:35.75390334 +0000 UTC m=+275.337288975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:33.825128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:33.825095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg" event={"ID":"5ac92020-4c0b-4ad2-85b0-aecd0dea69de","Type":"ContainerStarted","Data":"22555ba9b3c55fcb2d016c70306c3fddaae0817cf1daf71d8016bdfdd3cb0bf3"}
Sep 18 06:26:33.825572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:33.825184    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg" event={"ID":"5ac92020-4c0b-4ad2-85b0-aecd0dea69de","Type":"ContainerStarted","Data":"bad9830e914f0905aa539e62411edb996c72c020dcf12633029400fdf6d9211e"}
Sep 18 06:26:33.840531 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:33.840481    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-jgzjg" podStartSLOduration=1.778861906 podStartE2EDuration="2.840467462s" totalImagesPullingTime="1.061605556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:33.838724459 +0000 UTC m=+273.422110112" watchObservedRunningTime="2026-09-18 06:26:33.840467462 +0000 UTC m=+273.423853114"
Sep 18 06:26:34.721475 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:34.721449    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lflsh_6d5ac4a2-e29f-4c7f-b9db-d8c021a09487/dns-node-resolver/0.log"
Sep 18 06:26:35.723112 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:35.723085    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fqclk_f7f1513d-e978-4051-9a08-56ad33a5e160/node-ca/0.log"
Sep 18 06:26:35.770778 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:35.770745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:35.770892 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:35.770783    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:35.770892 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:35.770875    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 06:26:35.770963 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:35.770935    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:35.770963 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:35.770946    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:39.770926891 +0000 UTC m=+279.354312537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : configmap "kube-rbac-proxy" not found
Sep 18 06:26:35.771029 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:35.770969    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:39.770959345 +0000 UTC m=+279.354344975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:36.521510 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:36.521473    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhs6h_9343c376-e916-43cb-97fd-3002665a86d3/serve-healthcheck-canary/0.log"
Sep 18 06:26:36.922341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:36.922251    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-jgzjg_5ac92020-4c0b-4ad2-85b0-aecd0dea69de/migrator/0.log"
Sep 18 06:26:37.121383 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:37.121356    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-jgzjg_5ac92020-4c0b-4ad2-85b0-aecd0dea69de/graceful-termination/0.log"
Sep 18 06:26:37.320054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:37.320016    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:37.320265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:37.320069    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:37.320501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:37.320481    2567 scope.go:117] "RemoveContainer" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"
Sep 18 06:26:37.320682 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:37.320659    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:26:39.804906 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:39.804872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:39.804906 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:39.804908    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:39.805336 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:39.804998    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 06:26:39.805336 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:39.805031    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:39.805336 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:39.805068    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:47.805048675 +0000 UTC m=+287.388434329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : configmap "kube-rbac-proxy" not found
Sep 18 06:26:39.805336 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:39.805083    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:26:47.805077567 +0000 UTC m=+287.388463197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:41.526811 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:41.526766    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1"
Sep 18 06:26:41.526811 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:41.526766    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7cjhw" podUID="29bfe3a4-5b99-4e12-8574-e2bc18699d64"
Sep 18 06:26:41.526811 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:41.526766    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" podUID="8c8c6630-d359-48a6-b8c4-057493ecd834"
Sep 18 06:26:41.850105 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:41.850017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:26:41.850306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:41.850019    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:26:41.850306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:41.850025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:26:45.346487 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.346442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:26:45.346878 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.346504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:26:45.346878 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.346531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:26:45.346878 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:45.346615    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 18 06:26:45.346878 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:45.346690    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert podName:8c8c6630-d359-48a6-b8c4-057493ecd834 nodeName:}" failed. No retries permitted until 2026-09-18 06:28:47.346672537 +0000 UTC m=+406.930058181 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-gctpq" (UID: "8c8c6630-d359-48a6-b8c4-057493ecd834") : secret "networking-console-plugin-cert" not found
Sep 18 06:26:45.348797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.348773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/29bfe3a4-5b99-4e12-8574-e2bc18699d64-metrics-tls\") pod \"dns-default-7cjhw\" (UID: \"29bfe3a4-5b99-4e12-8574-e2bc18699d64\") " pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:26:45.348881 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.348861    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"image-registry-68d84f4889-wggh4\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") " pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:26:45.454741 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.454711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d7llw\""
Sep 18 06:26:45.454741 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.454711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sn4gt\""
Sep 18 06:26:45.461834 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.461815    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:26:45.461921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.461905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:26:45.741433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.741400    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cjhw"]
Sep 18 06:26:45.743708 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.743688    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:26:45.745219 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:26:45.745193    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bfe3a4_5b99_4e12_8574_e2bc18699d64.slice/crio-f2460746729a9df799b8736ff5e2cfbf6c2a818ce17e0d212b07c5417390bf89 WatchSource:0}: Error finding container f2460746729a9df799b8736ff5e2cfbf6c2a818ce17e0d212b07c5417390bf89: Status 404 returned error can't find the container with id f2460746729a9df799b8736ff5e2cfbf6c2a818ce17e0d212b07c5417390bf89
Sep 18 06:26:45.746306 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:26:45.746279    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb144779_35ac_443b_8c3b_0fa81bf95ca1.slice/crio-fbf128e12cf5d482c299dd06236f9a8a6083f759876b9b87233e487cf80d3d6b WatchSource:0}: Error finding container fbf128e12cf5d482c299dd06236f9a8a6083f759876b9b87233e487cf80d3d6b: Status 404 returned error can't find the container with id fbf128e12cf5d482c299dd06236f9a8a6083f759876b9b87233e487cf80d3d6b
Sep 18 06:26:45.865807 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.865766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" event={"ID":"cb144779-35ac-443b-8c3b-0fa81bf95ca1","Type":"ContainerStarted","Data":"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"}
Sep 18 06:26:45.865962 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.865920    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:26:45.865962 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.865958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" event={"ID":"cb144779-35ac-443b-8c3b-0fa81bf95ca1","Type":"ContainerStarted","Data":"fbf128e12cf5d482c299dd06236f9a8a6083f759876b9b87233e487cf80d3d6b"}
Sep 18 06:26:45.866979 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.866958    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cjhw" event={"ID":"29bfe3a4-5b99-4e12-8574-e2bc18699d64","Type":"ContainerStarted","Data":"f2460746729a9df799b8736ff5e2cfbf6c2a818ce17e0d212b07c5417390bf89"}
Sep 18 06:26:45.885541 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:45.885496    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" podStartSLOduration=262.885481205 podStartE2EDuration="4m22.885481205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:45.884941677 +0000 UTC m=+285.468327342" watchObservedRunningTime="2026-09-18 06:26:45.885481205 +0000 UTC m=+285.468866858"
Sep 18 06:26:47.867107 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:47.867083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:47.867537 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:47.867250    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:47.867537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:47.867267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:47.867537 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:47.867307    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls podName:9c27008c-d284-4b5e-8575-a368c862ec86 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:03.867288006 +0000 UTC m=+303.450673641 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-45psb" (UID: "9c27008c-d284-4b5e-8575-a368c862ec86") : secret "insights-runtime-extractor-tls" not found
Sep 18 06:26:47.867905 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:47.867664    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c27008c-d284-4b5e-8575-a368c862ec86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:26:47.874680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:47.874657    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cjhw" event={"ID":"29bfe3a4-5b99-4e12-8574-e2bc18699d64","Type":"ContainerStarted","Data":"d42a643139c93e996802280ac7ce4625360c30369333c7cc7d294161d5ca885f"}
Sep 18 06:26:48.879202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:48.879166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cjhw" event={"ID":"29bfe3a4-5b99-4e12-8574-e2bc18699d64","Type":"ContainerStarted","Data":"edee5b4f98ab6456c4dc545e04549e551276879f1fbc1fb35387c9c3c5c15296"}
Sep 18 06:26:48.879662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:48.879285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:26:48.895631 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:48.895588    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7cjhw" podStartSLOduration=251.864461895 podStartE2EDuration="4m13.895576294s" totalImagesPullingTime="2.031114399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:26:48.894563157 +0000 UTC m=+288.477948813" watchObservedRunningTime="2026-09-18 06:26:48.895576294 +0000 UTC m=+288.478961947"
Sep 18 06:26:51.903381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:51.903353    2567 scope.go:117] "RemoveContainer" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"
Sep 18 06:26:52.893432 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.893406    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:26:52.893844 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.893829    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/1.log"
Sep 18 06:26:52.893901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.893861    2567 generic.go:356] "Generic (PLEG): container finished" podID="4327d4f3-0131-4777-b6d5-df915eab4b4e" containerID="b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d" exitCode=255
Sep 18 06:26:52.893901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.893886    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" event={"ID":"4327d4f3-0131-4777-b6d5-df915eab4b4e","Type":"ContainerDied","Data":"b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d"}
Sep 18 06:26:52.893992 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.893927    2567 scope.go:117] "RemoveContainer" containerID="125857e6e789976960b66ada4df48676a2eeda0535df91c5ec3c61d15a52cbe5"
Sep 18 06:26:52.894256 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:52.894239    2567 scope.go:117] "RemoveContainer" containerID="b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d"
Sep 18 06:26:52.894433 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:52.894413    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:26:53.898828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:53.898798    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:26:54.904847 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:54.904807    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fwqdd" podUID="7f327502-c45b-4c98-802a-a4ad54e6fbfa"
Sep 18 06:26:57.320017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:57.319970    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:57.320428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:57.320044    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:26:57.320428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:57.320367    2567 scope.go:117] "RemoveContainer" containerID="b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d"
Sep 18 06:26:57.320531 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:26:57.320511    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:26:58.884045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:26:58.884016    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7cjhw"
Sep 18 06:27:00.806973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:00.806943    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:27:00.807569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:00.807548    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:27:00.817654 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:00.817632    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 06:27:01.503063 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.503035    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"]
Sep 18 06:27:01.505908 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.505894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:01.508260 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.508237    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 06:27:01.508364 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.508273    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zg6xg\""
Sep 18 06:27:01.513028 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.513004    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"]
Sep 18 06:27:01.573377 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.573339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a467d06-ba49-4234-aa8d-5bf7ea956240-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-svwhs\" (UID: \"2a467d06-ba49-4234-aa8d-5bf7ea956240\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:01.674576 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.674545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a467d06-ba49-4234-aa8d-5bf7ea956240-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-svwhs\" (UID: \"2a467d06-ba49-4234-aa8d-5bf7ea956240\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:01.676886 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.676864    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2a467d06-ba49-4234-aa8d-5bf7ea956240-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-svwhs\" (UID: \"2a467d06-ba49-4234-aa8d-5bf7ea956240\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:01.814837 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:01.814739    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:02.080482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:02.080396    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"]
Sep 18 06:27:02.083053 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:27:02.083028    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a467d06_ba49_4234_aa8d_5bf7ea956240.slice/crio-eb4ef548cff4a2dc45be0f360efbd43801de8b236baa0cdae3669b53d0c02c7d WatchSource:0}: Error finding container eb4ef548cff4a2dc45be0f360efbd43801de8b236baa0cdae3669b53d0c02c7d: Status 404 returned error can't find the container with id eb4ef548cff4a2dc45be0f360efbd43801de8b236baa0cdae3669b53d0c02c7d
Sep 18 06:27:02.085111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:02.085095    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:27:02.928465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:02.928431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs" event={"ID":"2a467d06-ba49-4234-aa8d-5bf7ea956240","Type":"ContainerStarted","Data":"eb4ef548cff4a2dc45be0f360efbd43801de8b236baa0cdae3669b53d0c02c7d"}
Sep 18 06:27:03.895267 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.895230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:27:03.897552 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.897523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c27008c-d284-4b5e-8575-a368c862ec86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-45psb\" (UID: \"9c27008c-d284-4b5e-8575-a368c862ec86\") " pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:27:03.933423 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.933385    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs" event={"ID":"2a467d06-ba49-4234-aa8d-5bf7ea956240","Type":"ContainerStarted","Data":"038a04111c9b3b35ed4c7dade5fa6afee1fe60d7cece0a259dc43e3e172c9363"}
Sep 18 06:27:03.933854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.933650    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:03.938059 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.938033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs"
Sep 18 06:27:03.948552 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:03.948505    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-svwhs" podStartSLOduration=1.770079944 podStartE2EDuration="2.948479133s" totalImagesPullingTime="1.178399189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:03.946805003 +0000 UTC m=+303.530190656" watchObservedRunningTime="2026-09-18 06:27:03.948479133 +0000 UTC m=+303.531864822"
Sep 18 06:27:04.063317 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.063285    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6ts\""
Sep 18 06:27:04.071287 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.071265    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-45psb"
Sep 18 06:27:04.337441 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.337405    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-45psb"]
Sep 18 06:27:04.340065 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:27:04.340032    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c27008c_d284_4b5e_8575_a368c862ec86.slice/crio-71421b247d3fc7fe8dae3187769aa067781235d8e4e8b0fb0b59cf47eac2b953 WatchSource:0}: Error finding container 71421b247d3fc7fe8dae3187769aa067781235d8e4e8b0fb0b59cf47eac2b953: Status 404 returned error can't find the container with id 71421b247d3fc7fe8dae3187769aa067781235d8e4e8b0fb0b59cf47eac2b953
Sep 18 06:27:04.555909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.555875    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"]
Sep 18 06:27:04.559183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.559162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.561883 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.561843    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 06:27:04.561996 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.561927    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 06:27:04.561996 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.561949    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 06:27:04.561996 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.561951    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 06:27:04.561996 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.561983    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 06:27:04.563082 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.563067    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-254nv\""
Sep 18 06:27:04.568462 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.568440    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"]
Sep 18 06:27:04.703288 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.703258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppq5\" (UniqueName: \"kubernetes.io/projected/3a8dbbed-3a71-404f-a4ad-2392eff47de3-kube-api-access-vppq5\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.703435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.703337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8dbbed-3a71-404f-a4ad-2392eff47de3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.703435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.703372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.703435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.703404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.803764 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.803723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppq5\" (UniqueName: \"kubernetes.io/projected/3a8dbbed-3a71-404f-a4ad-2392eff47de3-kube-api-access-vppq5\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.803913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.803788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8dbbed-3a71-404f-a4ad-2392eff47de3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.803913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.803820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.803913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.803857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.804040 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:04.803945    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 18 06:27:04.804040 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:04.804016    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls podName:3a8dbbed-3a71-404f-a4ad-2392eff47de3 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:05.303995847 +0000 UTC m=+304.887381494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-d594v" (UID: "3a8dbbed-3a71-404f-a4ad-2392eff47de3") : secret "prometheus-operator-tls" not found
Sep 18 06:27:04.804471 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.804451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a8dbbed-3a71-404f-a4ad-2392eff47de3-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.806081 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.806062    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.811668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.811639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppq5\" (UniqueName: \"kubernetes.io/projected/3a8dbbed-3a71-404f-a4ad-2392eff47de3-kube-api-access-vppq5\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:04.937879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.937854    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45psb" event={"ID":"9c27008c-d284-4b5e-8575-a368c862ec86","Type":"ContainerStarted","Data":"6d4091f665e8952a5e83efd1b9979146886d1ecb9a13b34208f124ad59298218"}
Sep 18 06:27:04.938222 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:04.937902    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45psb" event={"ID":"9c27008c-d284-4b5e-8575-a368c862ec86","Type":"ContainerStarted","Data":"71421b247d3fc7fe8dae3187769aa067781235d8e4e8b0fb0b59cf47eac2b953"}
Sep 18 06:27:05.306963 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.306877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:05.309364 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.309339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a8dbbed-3a71-404f-a4ad-2392eff47de3-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-d594v\" (UID: \"3a8dbbed-3a71-404f-a4ad-2392eff47de3\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:05.467622 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.467582    2567 patch_prober.go:28] interesting pod/image-registry-68d84f4889-wggh4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 06:27:05.467799 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.467647    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 06:27:05.468632 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.468611    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"
Sep 18 06:27:05.745436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.745407    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-d594v"]
Sep 18 06:27:05.748368 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:27:05.748337    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8dbbed_3a71_404f_a4ad_2392eff47de3.slice/crio-dfc427697b3b7960ab9897334117d1c1b08a0a011d5f929edde5ad7a8fb7727f WatchSource:0}: Error finding container dfc427697b3b7960ab9897334117d1c1b08a0a011d5f929edde5ad7a8fb7727f: Status 404 returned error can't find the container with id dfc427697b3b7960ab9897334117d1c1b08a0a011d5f929edde5ad7a8fb7727f
Sep 18 06:27:05.947740 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.947700    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45psb" event={"ID":"9c27008c-d284-4b5e-8575-a368c862ec86","Type":"ContainerStarted","Data":"56a0cad64eff1801910f097dfa129c7302cf464b7ae8a1be7a4479de67533f5c"}
Sep 18 06:27:05.948986 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:05.948953    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v" event={"ID":"3a8dbbed-3a71-404f-a4ad-2392eff47de3","Type":"ContainerStarted","Data":"dfc427697b3b7960ab9897334117d1c1b08a0a011d5f929edde5ad7a8fb7727f"}
Sep 18 06:27:06.875327 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:06.875294    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:27:06.954549 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:06.954504    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-45psb" event={"ID":"9c27008c-d284-4b5e-8575-a368c862ec86","Type":"ContainerStarted","Data":"ad04d65cd2c656e4cae3a9214039aab9e10c81f8b51b4db0a81519fbce4f2547"}
Sep 18 06:27:06.971349 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:06.971308    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-45psb" podStartSLOduration=34.203493396 podStartE2EDuration="35.971289974s" totalImagesPullingTime="1.767796578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 06:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:06.970638737 +0000 UTC m=+306.554024394" watchObservedRunningTime="2026-09-18 06:27:06.971289974 +0000 UTC m=+306.554675671"
Sep 18 06:27:07.903626 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:07.903595    2567 scope.go:117] "RemoveContainer" containerID="b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d"
Sep 18 06:27:07.903799 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:07.903742    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-kggmd_openshift-console-operator(4327d4f3-0131-4777-b6d5-df915eab4b4e)\"" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podUID="4327d4f3-0131-4777-b6d5-df915eab4b4e"
Sep 18 06:27:07.959188 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:07.959151    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v" event={"ID":"3a8dbbed-3a71-404f-a4ad-2392eff47de3","Type":"ContainerStarted","Data":"978e48a4aec95cd1b12cfbbaa21795973b09143b0e299d8e4cd353b6cabf50e6"}
Sep 18 06:27:07.959546 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:07.959217    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v" event={"ID":"3a8dbbed-3a71-404f-a4ad-2392eff47de3","Type":"ContainerStarted","Data":"35ef5db8b1843fa9b03de63c8482229b6bcfd8e97304464c048b21d03b9a3043"}
Sep 18 06:27:07.974188 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:07.974123    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-d594v" podStartSLOduration=2.7241817409999998 podStartE2EDuration="3.974108258s" totalImagesPullingTime="1.249926517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:07.973357511 +0000 UTC m=+307.556743164" watchObservedRunningTime="2026-09-18 06:27:07.974108258 +0000 UTC m=+307.557493908"
Sep 18 06:27:08.903369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:08.903330    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:27:09.929657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.929619    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lzkr6"]
Sep 18 06:27:09.932912 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.932894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:09.935413 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.935393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 06:27:09.935507 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.935457    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cmnll\""
Sep 18 06:27:09.935507 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.935456    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 06:27:09.935657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:09.935640    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 06:27:10.042034 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-root\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042034 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-textfile\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-wtmp\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-metrics-client-ca\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042234    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042295    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xndr\" (UniqueName: \"kubernetes.io/projected/d5f06a2a-b1fd-4229-ba17-7416e1894094-kube-api-access-5xndr\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042443 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042443 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042378    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-sys\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.042443 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.042396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143514 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-sys\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143514 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143710 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143590    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-sys\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143710 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-root\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143710 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-textfile\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-wtmp\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143733    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-root\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-metrics-client-ca\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143995 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143941    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-wtmp\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.143995 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.143968    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xndr\" (UniqueName: \"kubernetes.io/projected/d5f06a2a-b1fd-4229-ba17-7416e1894094-kube-api-access-5xndr\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.144087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.144027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.144087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.144026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-textfile\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.144214 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:10.144089    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 06:27:10.144214 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:10.144166    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls podName:d5f06a2a-b1fd-4229-ba17-7416e1894094 nodeName:}" failed. No retries permitted until 2026-09-18 06:27:10.644128597 +0000 UTC m=+310.227514244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls") pod "node-exporter-lzkr6" (UID: "d5f06a2a-b1fd-4229-ba17-7416e1894094") : secret "node-exporter-tls" not found
Sep 18 06:27:10.144442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.144421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-metrics-client-ca\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.144633 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.144616    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-accelerators-collector-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.145895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.145874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.153885 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.153865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xndr\" (UniqueName: \"kubernetes.io/projected/d5f06a2a-b1fd-4229-ba17-7416e1894094-kube-api-access-5xndr\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.649196 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.649160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.651430 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.651412    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5f06a2a-b1fd-4229-ba17-7416e1894094-node-exporter-tls\") pod \"node-exporter-lzkr6\" (UID: \"d5f06a2a-b1fd-4229-ba17-7416e1894094\") " pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.841094 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.841060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lzkr6"
Sep 18 06:27:10.854023 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:27:10.853995    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f06a2a_b1fd_4229_ba17_7416e1894094.slice/crio-90178c04c8a197611a26af29f4568f60032723085e6d991194466b1dfecefdcb WatchSource:0}: Error finding container 90178c04c8a197611a26af29f4568f60032723085e6d991194466b1dfecefdcb: Status 404 returned error can't find the container with id 90178c04c8a197611a26af29f4568f60032723085e6d991194466b1dfecefdcb
Sep 18 06:27:10.969840 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:10.969798    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzkr6" event={"ID":"d5f06a2a-b1fd-4229-ba17-7416e1894094","Type":"ContainerStarted","Data":"90178c04c8a197611a26af29f4568f60032723085e6d991194466b1dfecefdcb"}
Sep 18 06:27:12.977780 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:12.977747    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5f06a2a-b1fd-4229-ba17-7416e1894094" containerID="1e3c5d95aa5f2e69bd86c058dd5cd33dd15f0ed701a26d63ce34f2111a9fdc74" exitCode=0
Sep 18 06:27:12.978269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:12.977794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzkr6" event={"ID":"d5f06a2a-b1fd-4229-ba17-7416e1894094","Type":"ContainerDied","Data":"1e3c5d95aa5f2e69bd86c058dd5cd33dd15f0ed701a26d63ce34f2111a9fdc74"}
Sep 18 06:27:13.983442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:13.983406    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzkr6" event={"ID":"d5f06a2a-b1fd-4229-ba17-7416e1894094","Type":"ContainerStarted","Data":"e3a7d73331aa46626c1898605a4d257103b07f0f8012a14353298ff870ae2f64"}
Sep 18 06:27:13.983919 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:13.983456    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lzkr6" event={"ID":"d5f06a2a-b1fd-4229-ba17-7416e1894094","Type":"ContainerStarted","Data":"014de7f9f28ef9c9c5d2e4c6ba1461ce13de622bc33180a40aa0a4b35e497330"}
Sep 18 06:27:14.003068 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:14.003021    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lzkr6" podStartSLOduration=3.874916636 podStartE2EDuration="5.00300691s" totalImagesPullingTime="1.06711026s" totalInitContainerRuntime="60.980014ms" isStatefulPod=true podCreationTimestamp="2026-09-18 06:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:14.002324346 +0000 UTC m=+313.585710000" watchObservedRunningTime="2026-09-18 06:27:14.00300691 +0000 UTC m=+313.586392563"
Sep 18 06:27:16.058517 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.058477    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:27:16.062663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.062640    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.066032 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066003    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 06:27:16.066032 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066026    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 06:27:16.066237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 06:27:16.066237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066026    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 06:27:16.066394 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066368    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wdkns\""
Sep 18 06:27:16.066394 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066392    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 06:27:16.066521 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.066394    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 06:27:16.067436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067236    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 06:27:16.067436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067276    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 06:27:16.067436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067297    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 06:27:16.067642 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067445    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 06:27:16.067642 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067468    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 06:27:16.067885 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.067868    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 06:27:16.068028 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.068010    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ajk744t407pjr\""
Sep 18 06:27:16.068100 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.068042    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 06:27:16.070227 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.070207    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 06:27:16.073600 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.073579    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 06:27:16.079806 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.079783    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:27:16.193252 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193330    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193368    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193428 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqsz\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193517    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193567    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.193825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.193743    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294407 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294407 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294409    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294734    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294799    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqsz\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.294988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.294972    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.295583 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.295277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.295856 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.295828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.295944 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.295893    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.295944 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.295283    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.298462 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.297495    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.298927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.298902    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299016 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.298979    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299218 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299356 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299600 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299706 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299782 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299714    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.299894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.299850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.300547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.300522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.300919 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.300901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.301969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.301950    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.302283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.302241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.305513 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.305492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqsz\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz\") pod \"prometheus-k8s-0\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.375268 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.375181    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:16.668898 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.668871    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:27:16.670232 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:27:16.670201    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9b8ffe_cbd3_46c2_8e2a_e4128244f8f6.slice/crio-552cbb19dea5149729b15d2669257721c27596ed42850a55e706be114584ce8b WatchSource:0}: Error finding container 552cbb19dea5149729b15d2669257721c27596ed42850a55e706be114584ce8b: Status 404 returned error can't find the container with id 552cbb19dea5149729b15d2669257721c27596ed42850a55e706be114584ce8b
Sep 18 06:27:16.994372 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:16.994337    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"552cbb19dea5149729b15d2669257721c27596ed42850a55e706be114584ce8b"}
Sep 18 06:27:17.998861 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:17.998826    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9" exitCode=0
Sep 18 06:27:17.999234 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:17.998913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"}
Sep 18 06:27:18.601931 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:18.601895    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:27:22.021623 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:22.021578    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"}
Sep 18 06:27:22.022062 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:22.021716    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"}
Sep 18 06:27:22.909917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:22.909886    2567 scope.go:117] "RemoveContainer" containerID="b85ab2d149565ef5d0d751385d6ca20c19bece6a2dc9fc3f33e1d59d5e0ae30d"
Sep 18 06:27:23.029765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:23.029743    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:27:24.037719 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.037683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"}
Sep 18 06:27:24.038219 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.037760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"}
Sep 18 06:27:24.038219 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.037820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"}
Sep 18 06:27:24.038219 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.037859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerStarted","Data":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"}
Sep 18 06:27:24.039670 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.039650    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:27:24.039778 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.039695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" event={"ID":"4327d4f3-0131-4777-b6d5-df915eab4b4e","Type":"ContainerStarted","Data":"18e445a06cb84c92ca221a83b965d4c903133f39b929ab7ce68ba7f8362705a6"}
Sep 18 06:27:24.039973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.039953    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:27:24.045947 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.045928    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd"
Sep 18 06:27:24.068257 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.068213    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.231755571 podStartE2EDuration="8.06819452s" totalImagesPullingTime="5.82033574s" totalInitContainerRuntime="16.103209ms" isStatefulPod=false podCreationTimestamp="2026-09-18 06:27:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:24.066056032 +0000 UTC m=+323.649441712" watchObservedRunningTime="2026-09-18 06:27:24.06819452 +0000 UTC m=+323.651580174"
Sep 18 06:27:24.082797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:24.082745    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-kggmd" podStartSLOduration=55.716006337 podStartE2EDuration="58.082728864s" totalImagesPullingTime="2.366722527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:27:24.081949118 +0000 UTC m=+323.665334772" watchObservedRunningTime="2026-09-18 06:27:24.082728864 +0000 UTC m=+323.666114551"
Sep 18 06:27:26.376315 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:26.376281    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:27:36.085841 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:36.085803    2567 generic.go:356] "Generic (PLEG): container finished" podID="22febb06-4e87-4056-aa22-bd068da7eb94" containerID="2df0d9fc76566ce240d001cc697820f98c1cfa5cf06a7581a68893817b8ac56d" exitCode=0
Sep 18 06:27:36.086262 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:36.085876    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k" event={"ID":"22febb06-4e87-4056-aa22-bd068da7eb94","Type":"ContainerDied","Data":"2df0d9fc76566ce240d001cc697820f98c1cfa5cf06a7581a68893817b8ac56d"}
Sep 18 06:27:36.086262 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:36.086197    2567 scope.go:117] "RemoveContainer" containerID="2df0d9fc76566ce240d001cc697820f98c1cfa5cf06a7581a68893817b8ac56d"
Sep 18 06:27:36.972854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:36.972827    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhs6h_9343c376-e916-43cb-97fd-3002665a86d3/serve-healthcheck-canary/0.log"
Sep 18 06:27:37.091942 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:37.091908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-fhg9k" event={"ID":"22febb06-4e87-4056-aa22-bd068da7eb94","Type":"ContainerStarted","Data":"f5e4761cd868e2b4a72d7299d554c5a50d22b12e1edcb2c2b7ee48af8250166e"}
Sep 18 06:27:43.624663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.624596    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerName="registry" containerID="cri-o://016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9" gracePeriod=30
Sep 18 06:27:43.858246 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.858224    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:27:43.917215 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917289    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917329    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917407 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917363    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917444 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gtw7\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917444 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917437    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917544 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917461    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917544 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917510    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token\") pod \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\" (UID: \"cb144779-35ac-443b-8c3b-0fa81bf95ca1\") "
Sep 18 06:27:43.917892 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.917864    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:27:43.918197 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.918166    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:27:43.919748 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.919718    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:27:43.919853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.919791    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:27:43.919917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.919891    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:27:43.920054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.920033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:27:43.920260 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.920240    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7" (OuterVolumeSpecName: "kube-api-access-2gtw7") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "kube-api-access-2gtw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:27:43.925513 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:43.925491    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cb144779-35ac-443b-8c3b-0fa81bf95ca1" (UID: "cb144779-35ac-443b-8c3b-0fa81bf95ca1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:27:44.018599 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018564    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb144779-35ac-443b-8c3b-0fa81bf95ca1-ca-trust-extracted\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018599 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018592    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018599 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018600    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-registry-certificates\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018599 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018609    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtw7\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-kube-api-access-2gtw7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018848 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018618    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-installation-pull-secrets\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018848 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018626    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb144779-35ac-443b-8c3b-0fa81bf95ca1-trusted-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018848 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018634    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb144779-35ac-443b-8c3b-0fa81bf95ca1-bound-sa-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.018848 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.018642    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb144779-35ac-443b-8c3b-0fa81bf95ca1-image-registry-private-configuration\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:27:44.119402 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.119365    2567 generic.go:356] "Generic (PLEG): container finished" podID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerID="016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9" exitCode=0
Sep 18 06:27:44.119402 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.119403    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68d84f4889-wggh4"
Sep 18 06:27:44.119590 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.119451    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" event={"ID":"cb144779-35ac-443b-8c3b-0fa81bf95ca1","Type":"ContainerDied","Data":"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"}
Sep 18 06:27:44.119590 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.119492    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68d84f4889-wggh4" event={"ID":"cb144779-35ac-443b-8c3b-0fa81bf95ca1","Type":"ContainerDied","Data":"fbf128e12cf5d482c299dd06236f9a8a6083f759876b9b87233e487cf80d3d6b"}
Sep 18 06:27:44.119590 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.119511    2567 scope.go:117] "RemoveContainer" containerID="016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"
Sep 18 06:27:44.128034 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.128019    2567 scope.go:117] "RemoveContainer" containerID="016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"
Sep 18 06:27:44.128310 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:27:44.128290    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9\": container with ID starting with 016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9 not found: ID does not exist" containerID="016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"
Sep 18 06:27:44.128354 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.128317    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9"} err="failed to get container status \"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9\": rpc error: code = NotFound desc = could not find container \"016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9\": container with ID starting with 016208525f8d3438f2e852c2524fafea6e309ae6357d89e54edc2a22585bd6d9 not found: ID does not exist"
Sep 18 06:27:44.143117 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.139783    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:27:44.145795 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.145775    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68d84f4889-wggh4"]
Sep 18 06:27:44.908226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:27:44.908188    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" path="/var/lib/kubelet/pods/cb144779-35ac-443b-8c3b-0fa81bf95ca1/volumes"
Sep 18 06:28:14.764970 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:14.764923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:28:14.767211 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:14.767192    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f327502-c45b-4c98-802a-a4ad54e6fbfa-metrics-certs\") pod \"network-metrics-daemon-fwqdd\" (UID: \"7f327502-c45b-4c98-802a-a4ad54e6fbfa\") " pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:28:14.906104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:14.906074    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fg7cb\""
Sep 18 06:28:14.914217 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:14.914194    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fwqdd"
Sep 18 06:28:15.180789 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:15.180615    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fwqdd"]
Sep 18 06:28:15.183405 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:28:15.183377    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f327502_c45b_4c98_802a_a4ad54e6fbfa.slice/crio-82526617e280c4b901206fda9e73534eac0d97365f52dbdeb67dad853c0bae06 WatchSource:0}: Error finding container 82526617e280c4b901206fda9e73534eac0d97365f52dbdeb67dad853c0bae06: Status 404 returned error can't find the container with id 82526617e280c4b901206fda9e73534eac0d97365f52dbdeb67dad853c0bae06
Sep 18 06:28:15.232251 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:15.232213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwqdd" event={"ID":"7f327502-c45b-4c98-802a-a4ad54e6fbfa","Type":"ContainerStarted","Data":"82526617e280c4b901206fda9e73534eac0d97365f52dbdeb67dad853c0bae06"}
Sep 18 06:28:16.376223 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:16.376188    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:16.394852 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:16.394828    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:17.240910 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:17.240871    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwqdd" event={"ID":"7f327502-c45b-4c98-802a-a4ad54e6fbfa","Type":"ContainerStarted","Data":"28c087f61224a8799c90a349102d575edbcda13c2224af7ba525c03b8136c8a1"}
Sep 18 06:28:17.241089 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:17.240936    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fwqdd" event={"ID":"7f327502-c45b-4c98-802a-a4ad54e6fbfa","Type":"ContainerStarted","Data":"722bd97cf3f1e793d4cb97c6fbe5522a73d76467e47382622966430091d231c5"}
Sep 18 06:28:17.256265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:17.256221    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:17.257228 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:17.257186    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fwqdd" podStartSLOduration=375.26244702 podStartE2EDuration="6m16.25717107s" totalImagesPullingTime="994.72405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:28:17.25625919 +0000 UTC m=+376.839644845" watchObservedRunningTime="2026-09-18 06:28:17.25717107 +0000 UTC m=+376.840556722"
Sep 18 06:28:34.423918 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.423883    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:34.424486 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424207    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="prometheus" containerID="cri-o://ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9" gracePeriod=600
Sep 18 06:28:34.424486 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424258    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy" containerID="cri-o://e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f" gracePeriod=600
Sep 18 06:28:34.424486 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424268    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-web" containerID="cri-o://21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300" gracePeriod=600
Sep 18 06:28:34.424486 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424288    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="config-reloader" containerID="cri-o://4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea" gracePeriod=600
Sep 18 06:28:34.424486 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424299    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="thanos-sidecar" containerID="cri-o://31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433" gracePeriod=600
Sep 18 06:28:34.424738 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.424542    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d" gracePeriod=600
Sep 18 06:28:34.675311 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.675251    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:34.727096 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727066    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727342 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727323    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727435    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727549 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727536    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727627    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727727 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727714    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727824 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727811    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.727915 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.727903    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728020 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728008    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmqsz\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728110 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728099    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728236 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728223    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728326    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728408    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728515 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728505    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728619 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728708 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728787    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728883 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728872    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.728976 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.728966    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.729339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.729303    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.729896 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.729820    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.730157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.730116    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.731416 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.731387    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.731961 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.731933    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.732072 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.731964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:28:34.732158 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.732084    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.733784 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.733753    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.733883 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.733852    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config" (OuterVolumeSpecName: "config") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.734080 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.734037    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.734219 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.734112    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.734746 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.734715    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.734839 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.734818    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.735001 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.734979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.735692 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.735632    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out" (OuterVolumeSpecName: "config-out") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:28:34.735919 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.735712    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.736029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.736006    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:28:34.736445 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.736412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz" (OuterVolumeSpecName: "kube-api-access-cmqsz") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "kube-api-access-cmqsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:28:34.745279 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.745255    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config" (OuterVolumeSpecName: "web-config") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:28:34.830167 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830123    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle\") pod \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\" (UID: \"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6\") "
Sep 18 06:28:34.830334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830312    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830325    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-kube-rbac-proxy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830335    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830345    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-grpc-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830354    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-metrics-client-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830364    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830372    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830382    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830390    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830401    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqsz\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-kube-api-access-cmqsz\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830409    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-prometheus-k8s-db\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830420    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-metrics-client-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830434    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830445    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830454    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-config-out\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830447    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" (UID: "1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830462    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830495    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830506    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-web-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.830663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.830515    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-tls-assets\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:34.931367 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:34.931298    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:28:35.307440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307401    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d" exitCode=0
Sep 18 06:28:35.307440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307430    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f" exitCode=0
Sep 18 06:28:35.307440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307438    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300" exitCode=0
Sep 18 06:28:35.307440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307444    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433" exitCode=0
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307451    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea" exitCode=0
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307461    2567 generic.go:356] "Generic (PLEG): container finished" podID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9" exitCode=0
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307468    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307478    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"}
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307555    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"}
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307584    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"}
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"}
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307622    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307654    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"}
Sep 18 06:28:35.307705 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"}
Sep 18 06:28:35.308122 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.307735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6","Type":"ContainerDied","Data":"552cbb19dea5149729b15d2669257721c27596ed42850a55e706be114584ce8b"}
Sep 18 06:28:35.315240 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.315216    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.321833 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.321816    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.328574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.328549    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.330028 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.330010    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:35.334357 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.334338    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:35.335831 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.335812    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.342292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.342269    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.348864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.348843    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.355076 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355059    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.355350 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.355328    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.355431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355358    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.355431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355377    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.355640 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.355622    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.355676 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355646    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.355676 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355663    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.355928 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.355911    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.355973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355936    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.355973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.355957    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.356296 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.356278    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.356352 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356299    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.356352 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356312    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.356524 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.356508    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.356563 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356527    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.356563 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356539    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.356758 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.356742    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.356797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356762    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.356797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.356773    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.357008 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:35.356993    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.357055 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357011    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.357055 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357025    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.357357 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357332    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.357450 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357360    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.357682 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357655    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.357682 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.357683    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.358067 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358004    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.358067 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358031    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.358329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358310    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.358393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358329    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.358597 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358574    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.358671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358598    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.358859 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358839    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.358909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.358860    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.359084 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359069    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.359127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359085    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.359201 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359168    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:35.359332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359315    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.359332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359331    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.359462 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359446    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy"
Sep 18 06:28:35.359512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359466    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy"
Sep 18 06:28:35.359512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359479    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-web"
Sep 18 06:28:35.359512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359488    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-web"
Sep 18 06:28:35.359512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359499    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerName="registry"
Sep 18 06:28:35.359512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359507    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerName="registry"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359521    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-thanos"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359530    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-thanos"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359542    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="prometheus"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359551    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="prometheus"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359570    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="init-config-reloader"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359576    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="init-config-reloader"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359591    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="config-reloader"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359596    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="config-reloader"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359602    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="thanos-sidecar"
Sep 18 06:28:35.359658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359607    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="thanos-sidecar"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359671    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="thanos-sidecar"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359678    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-thanos"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359685    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb144779-35ac-443b-8c3b-0fa81bf95ca1" containerName="registry"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359690    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy-web"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359695    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="kube-rbac-proxy"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359700    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="prometheus"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359706    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" containerName="config-reloader"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359602    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359752    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359982    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.359997 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.359998    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.360420 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360224    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.360420 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360238    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.360498 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360443    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.360498 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360457    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.360701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360685    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.360701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360700    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.360931 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360917    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.360973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.360931    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.361115 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361099    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.361115 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361114    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.361352 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361332    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.361394 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361354    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.361667 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361643    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.361761 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361670    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.361956 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361940    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.361956 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.361955    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.362213 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362192    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.362291 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362214    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.362480 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362460    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.362568 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362482    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.362760 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362735    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.362851 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.362760    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.364817 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.364796    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.364817 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.364817    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.365093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365075    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.365093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365092    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.365347 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365329    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.365347 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365346    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.365592 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365571    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.365592 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365590    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.365854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365836    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.365893 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.365855    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.366079 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366056    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.366117 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366080    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.366361 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366347    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.366469 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366348    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.366469 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366464    2567 scope.go:117] "RemoveContainer" containerID="4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"
Sep 18 06:28:35.366860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366838    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d"} err="failed to get container status \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": rpc error: code = NotFound desc = could not find container \"4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d\": container with ID starting with 4ec69797348fbfe026ff32e70c09bd1f7d487b8d773fc51fdbf7ac27b157c58d not found: ID does not exist"
Sep 18 06:28:35.366860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.366861    2567 scope.go:117] "RemoveContainer" containerID="e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"
Sep 18 06:28:35.367126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367103    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f"} err="failed to get container status \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": rpc error: code = NotFound desc = could not find container \"e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f\": container with ID starting with e6fa9fcf7b916810c58fc72d8fe894efa16b3a65d99159b065568924e52eac3f not found: ID does not exist"
Sep 18 06:28:35.367202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367126    2567 scope.go:117] "RemoveContainer" containerID="21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"
Sep 18 06:28:35.367400 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367381    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300"} err="failed to get container status \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": rpc error: code = NotFound desc = could not find container \"21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300\": container with ID starting with 21e6523931c2ec6f70c1d3329c7f2a215c1e1f33c6f7682b82ee269591b53300 not found: ID does not exist"
Sep 18 06:28:35.367471 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367402    2567 scope.go:117] "RemoveContainer" containerID="31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"
Sep 18 06:28:35.367649 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367629    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433"} err="failed to get container status \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": rpc error: code = NotFound desc = could not find container \"31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433\": container with ID starting with 31bac256d674bfce58dccd9321272c0ef940b6a45cc76669598b66cd1f7bb433 not found: ID does not exist"
Sep 18 06:28:35.367698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367651    2567 scope.go:117] "RemoveContainer" containerID="4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"
Sep 18 06:28:35.367965 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367924    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea"} err="failed to get container status \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": rpc error: code = NotFound desc = could not find container \"4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea\": container with ID starting with 4f081fc0fbbbda344249057dfd38812f599c6ea004382ba37584f15540590dea not found: ID does not exist"
Sep 18 06:28:35.368029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.367965    2567 scope.go:117] "RemoveContainer" containerID="ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"
Sep 18 06:28:35.368350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368330    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9"} err="failed to get container status \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": rpc error: code = NotFound desc = could not find container \"ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9\": container with ID starting with ff966be03fbd6188c033e8aa6748a5a8d2e40499c2f99f3c8535a3dd04cf57a9 not found: ID does not exist"
Sep 18 06:28:35.368432 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368352    2567 scope.go:117] "RemoveContainer" containerID="cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"
Sep 18 06:28:35.368660 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368639    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9"} err="failed to get container status \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": rpc error: code = NotFound desc = could not find container \"cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9\": container with ID starting with cac4da21ce05def40f994b13381569532885b8b84bb95c60b346a2b435d571a9 not found: ID does not exist"
Sep 18 06:28:35.368749 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368690    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 06:28:35.368866 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368846    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 06:28:35.368998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368846    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 06:28:35.368998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368914    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 06:28:35.368998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.368933    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 06:28:35.369325 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.369294    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 06:28:35.369406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.369306    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 06:28:35.369461 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.369417    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ajk744t407pjr\""
Sep 18 06:28:35.369515 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.369502    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 06:28:35.369607 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.369594    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 06:28:35.370038 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.370021    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 06:28:35.370235 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.370219    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 06:28:35.370283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.370240    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wdkns\""
Sep 18 06:28:35.370328 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.370311    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 06:28:35.370448 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.370430    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 06:28:35.372390 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.372369    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 06:28:35.374770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.374753    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 06:28:35.383294 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.383274    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:35.435466 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435554    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435665    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhb9\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-kube-api-access-nvhb9\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435799    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435827    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.435858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435916    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.435950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.436318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.436014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.536853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.536853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536929    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.536995    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhb9\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-kube-api-access-nvhb9\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537047    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537094    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537185    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537305    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537711    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.537913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.537773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.538399 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.538374    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.538707 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.538684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540258    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540278    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.540652 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.540492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.541409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.541382    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.541496 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.541382    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.541631 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.541610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.541951 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.541912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.542323 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.542301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.543011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.542990    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.543375 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.543359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f34f2fd7-44aa-483c-a593-4cc50b728df3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.543799 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.543779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f34f2fd7-44aa-483c-a593-4cc50b728df3-config\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.544334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.544311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f34f2fd7-44aa-483c-a593-4cc50b728df3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.545878 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.545862    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhb9\" (UniqueName: \"kubernetes.io/projected/f34f2fd7-44aa-483c-a593-4cc50b728df3-kube-api-access-nvhb9\") pod \"prometheus-k8s-0\" (UID: \"f34f2fd7-44aa-483c-a593-4cc50b728df3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.675847 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.675743    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:35.955815 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:35.955784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 06:28:35.958383 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:28:35.958355    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34f2fd7_44aa_483c_a593_4cc50b728df3.slice/crio-36d4c2fa107e78f6f94031421d275dc052a60b66d960f8766e847f4c5f8f7d3e WatchSource:0}: Error finding container 36d4c2fa107e78f6f94031421d275dc052a60b66d960f8766e847f4c5f8f7d3e: Status 404 returned error can't find the container with id 36d4c2fa107e78f6f94031421d275dc052a60b66d960f8766e847f4c5f8f7d3e
Sep 18 06:28:36.313433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:36.313346    2567 generic.go:356] "Generic (PLEG): container finished" podID="f34f2fd7-44aa-483c-a593-4cc50b728df3" containerID="9df057bdccfe51b62ca7cee46081a4b89ec09e3fae85e32091d562e7431833df" exitCode=0
Sep 18 06:28:36.313569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:36.313436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerDied","Data":"9df057bdccfe51b62ca7cee46081a4b89ec09e3fae85e32091d562e7431833df"}
Sep 18 06:28:36.313569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:36.313510    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"36d4c2fa107e78f6f94031421d275dc052a60b66d960f8766e847f4c5f8f7d3e"}
Sep 18 06:28:36.909002 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:36.908918    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6" path="/var/lib/kubelet/pods/1d9b8ffe-cbd3-46c2-8e2a-e4128244f8f6/volumes"
Sep 18 06:28:37.322953 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.322920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"0257e3f5b4c9085feabdb1db0d8bf8f79381106e60689c811b47ed95aa9a4423"}
Sep 18 06:28:37.323175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.322983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"a363133e6a3d798792a5c52e66cfb5a05afd3b434cf94105267e1d9583a8e94c"}
Sep 18 06:28:37.323175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.323009    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"11ac53fd13f179c754888d314b4b03d38a709cecfb7e7a3e152bf6a34405c70b"}
Sep 18 06:28:37.323175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.323032    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"8db664c39c1b5fffb9a38ab0a4a9aa22973b9805c37518b685ce6d70b68f1581"}
Sep 18 06:28:37.323175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.323055    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"cc6ff8b5cc5d9d43595ebc85eab140af430bcfd8bd34c445e5694e03f2981196"}
Sep 18 06:28:37.323175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.323088    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f34f2fd7-44aa-483c-a593-4cc50b728df3","Type":"ContainerStarted","Data":"67ff0b4e8830f34670865fea999630d98da935ce437159ff6d2cf2282e125a15"}
Sep 18 06:28:37.362566 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:37.359268    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.340442658 podStartE2EDuration="2.359247357s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.804699ms" isStatefulPod=false podCreationTimestamp="2026-09-18 06:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:28:37.356119207 +0000 UTC m=+396.939504897" watchObservedRunningTime="2026-09-18 06:28:37.359247357 +0000 UTC m=+396.942633009"
Sep 18 06:28:40.675973 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:40.675939    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:28:44.850990 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:28:44.850939    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" podUID="8c8c6630-d359-48a6-b8c4-057493ecd834"
Sep 18 06:28:45.352191 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:45.352155    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:28:47.428252 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:47.428210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:28:47.430886 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:47.430850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8c8c6630-d359-48a6-b8c4-057493ecd834-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-gctpq\" (UID: \"8c8c6630-d359-48a6-b8c4-057493ecd834\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:28:47.455504 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:47.455474    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6bzkx\""
Sep 18 06:28:47.468231 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:47.468199    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"
Sep 18 06:28:47.738927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:47.738892    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-gctpq"]
Sep 18 06:28:47.742344 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:28:47.742308    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8c6630_d359_48a6_b8c4_057493ecd834.slice/crio-2403eea2154523e9a728a3c43b57b1de7818b5c82700b558ee10c3720f4bbaac WatchSource:0}: Error finding container 2403eea2154523e9a728a3c43b57b1de7818b5c82700b558ee10c3720f4bbaac: Status 404 returned error can't find the container with id 2403eea2154523e9a728a3c43b57b1de7818b5c82700b558ee10c3720f4bbaac
Sep 18 06:28:48.362879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:48.362844    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" event={"ID":"8c8c6630-d359-48a6-b8c4-057493ecd834","Type":"ContainerStarted","Data":"2403eea2154523e9a728a3c43b57b1de7818b5c82700b558ee10c3720f4bbaac"}
Sep 18 06:28:49.367678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:49.367639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" event={"ID":"8c8c6630-d359-48a6-b8c4-057493ecd834","Type":"ContainerStarted","Data":"6b1b75e4c15e1f4c3b900d26fce79e56fdb21ef6d7362b8ba61f542e3ff79f26"}
Sep 18 06:28:49.382746 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:28:49.382660    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-gctpq" podStartSLOduration=384.998686036 podStartE2EDuration="6m26.382645221s" totalImagesPullingTime="1.383959185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:28:49.381689023 +0000 UTC m=+408.965074678" watchObservedRunningTime="2026-09-18 06:28:49.382645221 +0000 UTC m=+408.966030874"
Sep 18 06:29:35.676948 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:29:35.676864    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:29:35.692043 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:29:35.692016    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:29:36.559176 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:29:36.559121    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 06:31:39.088495 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.088459    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"]
Sep 18 06:31:39.091544 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.091529    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.094192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.094169    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 06:31:39.094310 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.094205    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7vfhm\""
Sep 18 06:31:39.095189 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.095174    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 06:31:39.098900 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.098877    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"]
Sep 18 06:31:39.128421 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.128397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jwn\" (UniqueName: \"kubernetes.io/projected/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-kube-api-access-t7jwn\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.128517 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.128427    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.229571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.229524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jwn\" (UniqueName: \"kubernetes.io/projected/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-kube-api-access-t7jwn\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.229571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.229578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.229963 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.229941    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.237598 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.237566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jwn\" (UniqueName: \"kubernetes.io/projected/2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9-kube-api-access-t7jwn\") pod \"openshift-lws-operator-fd8ccff4c-jmznw\" (UID: \"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.409153 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.409056    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"
Sep 18 06:31:39.676727 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.676689    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw"]
Sep 18 06:31:39.679082 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:31:39.679056    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fac51fd_5f4e_4dc9_a611_93e1e74ca2d9.slice/crio-969016b2a8b8ad7a9ccf6d0244c3dea9943441a7046e22a86acf57b8264168c4 WatchSource:0}: Error finding container 969016b2a8b8ad7a9ccf6d0244c3dea9943441a7046e22a86acf57b8264168c4: Status 404 returned error can't find the container with id 969016b2a8b8ad7a9ccf6d0244c3dea9943441a7046e22a86acf57b8264168c4
Sep 18 06:31:39.979601 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:39.979563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw" event={"ID":"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9","Type":"ContainerStarted","Data":"969016b2a8b8ad7a9ccf6d0244c3dea9943441a7046e22a86acf57b8264168c4"}
Sep 18 06:31:42.993786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:42.993752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw" event={"ID":"2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9","Type":"ContainerStarted","Data":"2b92a96f166b71e3d7737649c5665a9e1db8d3aa837147bd0fda49b5d0c44dae"}
Sep 18 06:31:43.008667 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:43.008614    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jmznw" podStartSLOduration=1.623430959 podStartE2EDuration="4.008597524s" totalImagesPullingTime="2.385166565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:43.008128524 +0000 UTC m=+582.591514178" watchObservedRunningTime="2026-09-18 06:31:43.008597524 +0000 UTC m=+582.591983176"
Sep 18 06:31:53.562203 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.562165    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"]
Sep 18 06:31:53.565461 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.565442    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.569568 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.569533    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 06:31:53.569681 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.569577    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pjpgd\""
Sep 18 06:31:53.569681 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.569595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 06:31:53.569681 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.569626    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 06:31:53.574864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.574841    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"]
Sep 18 06:31:53.642959 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.642924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.643162 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.642968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5062625-380d-4384-af3a-037c8de0f9ac-manager-config\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.643162 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.642986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.643162 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.643098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcvx\" (UniqueName: \"kubernetes.io/projected/a5062625-380d-4384-af3a-037c8de0f9ac-kube-api-access-swcvx\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.744406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.744371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5062625-380d-4384-af3a-037c8de0f9ac-manager-config\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.744406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.744406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.744649 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.744453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcvx\" (UniqueName: \"kubernetes.io/projected/a5062625-380d-4384-af3a-037c8de0f9ac-kube-api-access-swcvx\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.744649 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.744491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.745048 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.745025    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5062625-380d-4384-af3a-037c8de0f9ac-manager-config\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.746891 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.746868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-metrics-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.746960 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.746913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5062625-380d-4384-af3a-037c8de0f9ac-cert\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.753005 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.752982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcvx\" (UniqueName: \"kubernetes.io/projected/a5062625-380d-4384-af3a-037c8de0f9ac-kube-api-access-swcvx\") pod \"lws-controller-manager-64c86f57d4-68vm7\" (UID: \"a5062625-380d-4384-af3a-037c8de0f9ac\") " pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:53.875106 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:53.875012    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:54.145295 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:54.145180    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"]
Sep 18 06:31:54.148199 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:31:54.148161    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5062625_380d_4384_af3a_037c8de0f9ac.slice/crio-2f253276aefe2b42e4f2ae59947a2fe4b1483decb585bb44785556fa182894b1 WatchSource:0}: Error finding container 2f253276aefe2b42e4f2ae59947a2fe4b1483decb585bb44785556fa182894b1: Status 404 returned error can't find the container with id 2f253276aefe2b42e4f2ae59947a2fe4b1483decb585bb44785556fa182894b1
Sep 18 06:31:55.039518 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:55.039480    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7" event={"ID":"a5062625-380d-4384-af3a-037c8de0f9ac","Type":"ContainerStarted","Data":"2f253276aefe2b42e4f2ae59947a2fe4b1483decb585bb44785556fa182894b1"}
Sep 18 06:31:56.045132 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:56.045041    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7" event={"ID":"a5062625-380d-4384-af3a-037c8de0f9ac","Type":"ContainerStarted","Data":"150ae383b9e536e252a3b38162eb42cd08403317d91e3a8db94f61d15d1a853c"}
Sep 18 06:31:56.045506 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:56.045193    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:31:56.060716 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:31:56.060670    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7" podStartSLOduration=1.447451067 podStartE2EDuration="3.060656412s" totalImagesPullingTime="1.613205345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:31:56.059730628 +0000 UTC m=+595.643116281" watchObservedRunningTime="2026-09-18 06:31:56.060656412 +0000 UTC m=+595.644042065"
Sep 18 06:32:00.833210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:00.833179    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:32:00.833860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:00.833840    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:32:02.401844 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.401808    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"]
Sep 18 06:32:02.404104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.404088    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.408733 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.408713    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 06:32:02.410058 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.410031    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zlgzp\""
Sep 18 06:32:02.410241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.410227    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 06:32:02.410754 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.410667    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 06:32:02.410754 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.410672    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 06:32:02.424791 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.424763    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"]
Sep 18 06:32:02.522621 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.522590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbfm\" (UniqueName: \"kubernetes.io/projected/1bbc1505-2348-44a4-a62c-1774b61f3136-kube-api-access-vlbfm\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.522804 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.522645    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.522804 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.522673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.522804 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.522694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.522804 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.522723    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623452 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623408    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbfm\" (UniqueName: \"kubernetes.io/projected/1bbc1505-2348-44a4-a62c-1774b61f3136-kube-api-access-vlbfm\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.623980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.623955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-manifests\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.625834 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.625813    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bbc1505-2348-44a4-a62c-1774b61f3136-tmp\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.625949 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.625894    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.625998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.625963    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bbc1505-2348-44a4-a62c-1774b61f3136-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.631305 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.631286    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbfm\" (UniqueName: \"kubernetes.io/projected/1bbc1505-2348-44a4-a62c-1774b61f3136-kube-api-access-vlbfm\") pod \"opendatahub-operator-controller-manager-6fd575c64f-4q4d8\" (UID: \"1bbc1505-2348-44a4-a62c-1774b61f3136\") " pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.715061 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.715015    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:02.990278 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.990181    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"]
Sep 18 06:32:02.992587 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:32:02.992546    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbc1505_2348_44a4_a62c_1774b61f3136.slice/crio-821b724f6190c421489820bf6ce2878a693b654537a7ceec73279d2d7fcf87ca WatchSource:0}: Error finding container 821b724f6190c421489820bf6ce2878a693b654537a7ceec73279d2d7fcf87ca: Status 404 returned error can't find the container with id 821b724f6190c421489820bf6ce2878a693b654537a7ceec73279d2d7fcf87ca
Sep 18 06:32:02.994440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:02.994420    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:32:03.071874 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:03.071836    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8" event={"ID":"1bbc1505-2348-44a4-a62c-1774b61f3136","Type":"ContainerStarted","Data":"821b724f6190c421489820bf6ce2878a693b654537a7ceec73279d2d7fcf87ca"}
Sep 18 06:32:07.051045 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:07.051018    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64c86f57d4-68vm7"
Sep 18 06:32:07.093489 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:07.093446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8" event={"ID":"1bbc1505-2348-44a4-a62c-1774b61f3136","Type":"ContainerStarted","Data":"fcd69093bb459aafce38b819fd5230d122cf2b2a8d56667e0bd85885a23713e9"}
Sep 18 06:32:08.099886 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:08.099853    2567 generic.go:356] "Generic (PLEG): container finished" podID="1bbc1505-2348-44a4-a62c-1774b61f3136" containerID="fcd69093bb459aafce38b819fd5230d122cf2b2a8d56667e0bd85885a23713e9" exitCode=0
Sep 18 06:32:08.100347 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:08.099929    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8" event={"ID":"1bbc1505-2348-44a4-a62c-1774b61f3136","Type":"ContainerDied","Data":"fcd69093bb459aafce38b819fd5230d122cf2b2a8d56667e0bd85885a23713e9"}
Sep 18 06:32:09.106429 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:09.106390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8" event={"ID":"1bbc1505-2348-44a4-a62c-1774b61f3136","Type":"ContainerStarted","Data":"bfa79ba8501bf20a867acb95a6c9875951b531bedc2d68af7a3b9df3a66e23b4"}
Sep 18 06:32:09.107227 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:09.107203    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:09.128795 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:09.128757    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8" podStartSLOduration=2.713314697 podStartE2EDuration="7.128739953s" totalImagesPullingTime="3.380787587s" totalInitContainerRuntime="1.034637669s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:09.127365818 +0000 UTC m=+608.710751471" watchObservedRunningTime="2026-09-18 06:32:09.128739953 +0000 UTC m=+608.712125607"
Sep 18 06:32:20.112609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:20.112576    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fd575c64f-4q4d8"
Sep 18 06:32:22.049917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.049883    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"]
Sep 18 06:32:22.058056 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.058036    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.061539 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.061517    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 06:32:22.062639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.062619    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 06:32:22.062924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.062898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 06:32:22.063059 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.062965    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tkjkp\""
Sep 18 06:32:22.063243 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.063226    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 06:32:22.064874 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.064850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"]
Sep 18 06:32:22.075809 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.075786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802501dc-0496-464c-bebf-9d640a6dc13f-tls-certs\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.075916 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.075813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/802501dc-0496-464c-bebf-9d640a6dc13f-tmp\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.075916 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.075897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7sqt\" (UniqueName: \"kubernetes.io/projected/802501dc-0496-464c-bebf-9d640a6dc13f-kube-api-access-f7sqt\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.177191 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.177130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7sqt\" (UniqueName: \"kubernetes.io/projected/802501dc-0496-464c-bebf-9d640a6dc13f-kube-api-access-f7sqt\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.177363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.177204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802501dc-0496-464c-bebf-9d640a6dc13f-tls-certs\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.177363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.177220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/802501dc-0496-464c-bebf-9d640a6dc13f-tmp\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.179425 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.179402    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/802501dc-0496-464c-bebf-9d640a6dc13f-tmp\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.179651 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.179630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802501dc-0496-464c-bebf-9d640a6dc13f-tls-certs\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.184371 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.184350    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7sqt\" (UniqueName: \"kubernetes.io/projected/802501dc-0496-464c-bebf-9d640a6dc13f-kube-api-access-f7sqt\") pod \"kube-auth-proxy-5bbd68dc59-wktc5\" (UID: \"802501dc-0496-464c-bebf-9d640a6dc13f\") " pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.369278 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.369204    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"
Sep 18 06:32:22.659532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:22.659459    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5"]
Sep 18 06:32:22.662535 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:32:22.662501    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802501dc_0496_464c_bebf_9d640a6dc13f.slice/crio-41c9101f36e57c3629ce701c6df0852a4fed935b7a36016d3e817c447f96e928 WatchSource:0}: Error finding container 41c9101f36e57c3629ce701c6df0852a4fed935b7a36016d3e817c447f96e928: Status 404 returned error can't find the container with id 41c9101f36e57c3629ce701c6df0852a4fed935b7a36016d3e817c447f96e928
Sep 18 06:32:23.163674 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:23.163637    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5" event={"ID":"802501dc-0496-464c-bebf-9d640a6dc13f","Type":"ContainerStarted","Data":"41c9101f36e57c3629ce701c6df0852a4fed935b7a36016d3e817c447f96e928"}
Sep 18 06:32:26.179966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:32:26.179932    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5" event={"ID":"802501dc-0496-464c-bebf-9d640a6dc13f","Type":"ContainerStarted","Data":"2196635ba858d541830015614e20eda137b9b8f307ca9623e4aff5b24bb74226"}
Sep 18 06:33:57.721037 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.720911    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bbd68dc59-wktc5" podStartSLOduration=92.880386357 podStartE2EDuration="1m35.720886345s" totalImagesPullingTime="2.840499988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:32:26.194796624 +0000 UTC m=+625.778182278" watchObservedRunningTime="2026-09-18 06:33:57.720886345 +0000 UTC m=+717.304271998"
Sep 18 06:33:57.722459 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.722439    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"]
Sep 18 06:33:57.724597 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.724579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:33:57.727521 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.727504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 06:33:57.727739 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.727722    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sl75v\""
Sep 18 06:33:57.727820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.727722    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 06:33:57.727875 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.727728    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 06:33:57.737047 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.737026    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"]
Sep 18 06:33:57.863075 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.863034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qlb\" (UniqueName: \"kubernetes.io/projected/0540e07a-d58f-442e-881a-051f9726ff3c-kube-api-access-s7qlb\") pod \"dns-operator-controller-manager-7b84cd7c9c-xvrjd\" (UID: \"0540e07a-d58f-442e-881a-051f9726ff3c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:33:57.964574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.964524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qlb\" (UniqueName: \"kubernetes.io/projected/0540e07a-d58f-442e-881a-051f9726ff3c-kube-api-access-s7qlb\") pod \"dns-operator-controller-manager-7b84cd7c9c-xvrjd\" (UID: \"0540e07a-d58f-442e-881a-051f9726ff3c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:33:57.972510 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:57.972449    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qlb\" (UniqueName: \"kubernetes.io/projected/0540e07a-d58f-442e-881a-051f9726ff3c-kube-api-access-s7qlb\") pod \"dns-operator-controller-manager-7b84cd7c9c-xvrjd\" (UID: \"0540e07a-d58f-442e-881a-051f9726ff3c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:33:58.033984 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:58.033951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:33:58.302539 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:58.302509    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"]
Sep 18 06:33:58.305072 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:33:58.305039    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0540e07a_d58f_442e_881a_051f9726ff3c.slice/crio-6220a085941d5006b23e20c0e9cb0fb93029cf74f53330357f1d1427b1ff3639 WatchSource:0}: Error finding container 6220a085941d5006b23e20c0e9cb0fb93029cf74f53330357f1d1427b1ff3639: Status 404 returned error can't find the container with id 6220a085941d5006b23e20c0e9cb0fb93029cf74f53330357f1d1427b1ff3639
Sep 18 06:33:58.549761 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:33:58.549732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd" event={"ID":"0540e07a-d58f-442e-881a-051f9726ff3c","Type":"ContainerStarted","Data":"6220a085941d5006b23e20c0e9cb0fb93029cf74f53330357f1d1427b1ff3639"}
Sep 18 06:34:00.817046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:00.817023    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 06:34:01.569057 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:01.569020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd" event={"ID":"0540e07a-d58f-442e-881a-051f9726ff3c","Type":"ContainerStarted","Data":"ae80f3a16efa33d71bc42c2a975601358066c0053bcad33d551ea3d74c9a6257"}
Sep 18 06:34:01.569234 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:01.569169    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:34:01.588007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:01.587956    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd" podStartSLOduration=2.080612836 podStartE2EDuration="4.587939587s" totalImagesPullingTime="2.507326751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:34:01.585566675 +0000 UTC m=+721.168952329" watchObservedRunningTime="2026-09-18 06:34:01.587939587 +0000 UTC m=+721.171325240"
Sep 18 06:34:02.672001 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.671967    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"]
Sep 18 06:34:02.674196 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.674178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.676651 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.676621    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 06:34:02.676770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.676664    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 06:34:02.676770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.676684    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p79c2\""
Sep 18 06:34:02.682668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.682647    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"]
Sep 18 06:34:02.807014 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.806975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/03070c95-7875-43df-96b5-48df54107303-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.807014 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.807019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxx7j\" (UniqueName: \"kubernetes.io/projected/03070c95-7875-43df-96b5-48df54107303-kube-api-access-vxx7j\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.807248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.807128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/03070c95-7875-43df-96b5-48df54107303-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.908230 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.908199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/03070c95-7875-43df-96b5-48df54107303-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.908230 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.908229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxx7j\" (UniqueName: \"kubernetes.io/projected/03070c95-7875-43df-96b5-48df54107303-kube-api-access-vxx7j\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.908433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.908291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/03070c95-7875-43df-96b5-48df54107303-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.908917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.908890    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/03070c95-7875-43df-96b5-48df54107303-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.910548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.910526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/03070c95-7875-43df-96b5-48df54107303-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.917484 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.917464    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxx7j\" (UniqueName: \"kubernetes.io/projected/03070c95-7875-43df-96b5-48df54107303-kube-api-access-vxx7j\") pod \"kuadrant-console-plugin-fbdd4f445-d89v9\" (UID: \"03070c95-7875-43df-96b5-48df54107303\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:02.984074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:02.984041    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"
Sep 18 06:34:03.249661 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:03.249474    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9"]
Sep 18 06:34:03.252184 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:34:03.252150    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03070c95_7875_43df_96b5_48df54107303.slice/crio-3c408ad0ca30e933e0f2c56dc6ce08e698bbb429e0e491d9cb6b400db9ecac3b WatchSource:0}: Error finding container 3c408ad0ca30e933e0f2c56dc6ce08e698bbb429e0e491d9cb6b400db9ecac3b: Status 404 returned error can't find the container with id 3c408ad0ca30e933e0f2c56dc6ce08e698bbb429e0e491d9cb6b400db9ecac3b
Sep 18 06:34:03.578634 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:03.578547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9" event={"ID":"03070c95-7875-43df-96b5-48df54107303","Type":"ContainerStarted","Data":"3c408ad0ca30e933e0f2c56dc6ce08e698bbb429e0e491d9cb6b400db9ecac3b"}
Sep 18 06:34:09.607652 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:09.607610    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9" event={"ID":"03070c95-7875-43df-96b5-48df54107303","Type":"ContainerStarted","Data":"2c20178c499a0f22cbbe1964b4bd734d40c1239682adaa2f5c6f9a5b6a57d5e4"}
Sep 18 06:34:09.623653 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:09.623604    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d89v9" podStartSLOduration=2.270943603 podStartE2EDuration="7.623589174s" totalImagesPullingTime="5.352645571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:34:09.622313114 +0000 UTC m=+729.205698771" watchObservedRunningTime="2026-09-18 06:34:09.623589174 +0000 UTC m=+729.206974848"
Sep 18 06:34:12.576030 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:12.575993    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xvrjd"
Sep 18 06:34:56.737421 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.737390    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:34:56.739487 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.739472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.743528 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.743508    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 06:34:56.750128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.750106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:34:56.830350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.830319    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:34:56.851354 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.851322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thn54\" (UniqueName: \"kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.851354 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.851356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.951862 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.951829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thn54\" (UniqueName: \"kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.951862 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.951867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.952560 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.952539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:56.959251 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:56.959219    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thn54\" (UniqueName: \"kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54\") pod \"limitador-limitador-8588749977-tfn6f\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") " pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:57.049253 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.049171    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:34:57.317608 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.317581    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:34:57.320364 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:34:57.320330    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa531efb_8c58_4f80_a567_7205699e11a8.slice/crio-9983e5c7273a79a2b244cc551cccc8d89d90f976077a2da109991e5cc1f84c1d WatchSource:0}: Error finding container 9983e5c7273a79a2b244cc551cccc8d89d90f976077a2da109991e5cc1f84c1d: Status 404 returned error can't find the container with id 9983e5c7273a79a2b244cc551cccc8d89d90f976077a2da109991e5cc1f84c1d
Sep 18 06:34:57.636659 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.636571    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:34:57.642774 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.641009    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:34:57.643953 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.643927    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zcv2z\""
Sep 18 06:34:57.645625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.645599    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:34:57.758678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.758642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tcgk\" (UniqueName: \"kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk\") pod \"authorino-59c974bd48-xnz85\" (UID: \"99903ab2-ab74-43ac-929a-3a4e54c32535\") " pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:34:57.814886 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.814850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" event={"ID":"aa531efb-8c58-4f80-a567-7205699e11a8","Type":"ContainerStarted","Data":"9983e5c7273a79a2b244cc551cccc8d89d90f976077a2da109991e5cc1f84c1d"}
Sep 18 06:34:57.860068 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.860036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcgk\" (UniqueName: \"kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk\") pod \"authorino-59c974bd48-xnz85\" (UID: \"99903ab2-ab74-43ac-929a-3a4e54c32535\") " pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:34:57.867843 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.867819    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcgk\" (UniqueName: \"kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk\") pod \"authorino-59c974bd48-xnz85\" (UID: \"99903ab2-ab74-43ac-929a-3a4e54c32535\") " pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:34:57.952080 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:57.952053    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:34:58.222623 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:58.222601    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:34:58.224679 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:34:58.224647    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99903ab2_ab74_43ac_929a_3a4e54c32535.slice/crio-7d0c563bea03dc3e732dd0446f1e04d241df63f5ca9209a8b499f01221d40486 WatchSource:0}: Error finding container 7d0c563bea03dc3e732dd0446f1e04d241df63f5ca9209a8b499f01221d40486: Status 404 returned error can't find the container with id 7d0c563bea03dc3e732dd0446f1e04d241df63f5ca9209a8b499f01221d40486
Sep 18 06:34:58.820341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:34:58.820304    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xnz85" event={"ID":"99903ab2-ab74-43ac-929a-3a4e54c32535","Type":"ContainerStarted","Data":"7d0c563bea03dc3e732dd0446f1e04d241df63f5ca9209a8b499f01221d40486"}
Sep 18 06:35:00.835212 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:00.835175    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" event={"ID":"aa531efb-8c58-4f80-a567-7205699e11a8","Type":"ContainerStarted","Data":"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"}
Sep 18 06:35:00.835648 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:00.835318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:35:00.850702 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:00.850657    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" podStartSLOduration=2.114742693 podStartE2EDuration="4.850641624s" totalImagesPullingTime="2.735898931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:35:00.850046433 +0000 UTC m=+780.433432096" watchObservedRunningTime="2026-09-18 06:35:00.850641624 +0000 UTC m=+780.434027277"
Sep 18 06:35:01.841103 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:01.841061    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xnz85" event={"ID":"99903ab2-ab74-43ac-929a-3a4e54c32535","Type":"ContainerStarted","Data":"be65d7dafd9d0119fb6f1c0108dfa3ab972cdd696b3b7e01f14d3d7cd7b3d285"}
Sep 18 06:35:01.855456 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:01.855389    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-xnz85" podStartSLOduration=2.079868209 podStartE2EDuration="4.855373979s" totalImagesPullingTime="2.77550577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:35:01.854910427 +0000 UTC m=+781.438296072" watchObservedRunningTime="2026-09-18 06:35:01.855373979 +0000 UTC m=+781.438759632"
Sep 18 06:35:11.636092 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:11.636052    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:35:11.636583 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:11.636333    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" podUID="aa531efb-8c58-4f80-a567-7205699e11a8" containerName="limitador" containerID="cri-o://3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd" gracePeriod=30
Sep 18 06:35:11.637028 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:11.636930    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:35:12.180049 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.180026    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:35:12.277248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.277161    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file\") pod \"aa531efb-8c58-4f80-a567-7205699e11a8\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") "
Sep 18 06:35:12.277509 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.277485    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file" (OuterVolumeSpecName: "config-file") pod "aa531efb-8c58-4f80-a567-7205699e11a8" (UID: "aa531efb-8c58-4f80-a567-7205699e11a8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:35:12.377695 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.377661    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thn54\" (UniqueName: \"kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54\") pod \"aa531efb-8c58-4f80-a567-7205699e11a8\" (UID: \"aa531efb-8c58-4f80-a567-7205699e11a8\") "
Sep 18 06:35:12.377850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.377839    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aa531efb-8c58-4f80-a567-7205699e11a8-config-file\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:12.379750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.379723    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54" (OuterVolumeSpecName: "kube-api-access-thn54") pod "aa531efb-8c58-4f80-a567-7205699e11a8" (UID: "aa531efb-8c58-4f80-a567-7205699e11a8"). InnerVolumeSpecName "kube-api-access-thn54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:35:12.478451 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.478423    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thn54\" (UniqueName: \"kubernetes.io/projected/aa531efb-8c58-4f80-a567-7205699e11a8-kube-api-access-thn54\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:12.888950 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.888913    2567 generic.go:356] "Generic (PLEG): container finished" podID="aa531efb-8c58-4f80-a567-7205699e11a8" containerID="3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd" exitCode=0
Sep 18 06:35:12.889369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.888962    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f"
Sep 18 06:35:12.889369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.888979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" event={"ID":"aa531efb-8c58-4f80-a567-7205699e11a8","Type":"ContainerDied","Data":"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"}
Sep 18 06:35:12.889369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.889021    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tfn6f" event={"ID":"aa531efb-8c58-4f80-a567-7205699e11a8","Type":"ContainerDied","Data":"9983e5c7273a79a2b244cc551cccc8d89d90f976077a2da109991e5cc1f84c1d"}
Sep 18 06:35:12.889369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.889043    2567 scope.go:117] "RemoveContainer" containerID="3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"
Sep 18 06:35:12.898920 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.898903    2567 scope.go:117] "RemoveContainer" containerID="3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"
Sep 18 06:35:12.899394 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:35:12.899326    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd\": container with ID starting with 3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd not found: ID does not exist" containerID="3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"
Sep 18 06:35:12.899394 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.899362    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd"} err="failed to get container status \"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd\": rpc error: code = NotFound desc = could not find container \"3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd\": container with ID starting with 3dc5b8cb20acccbe505f0cbcc7d359732d995673e0a044650eff48854ff125bd not found: ID does not exist"
Sep 18 06:35:12.909757 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.909734    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:35:12.913081 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:12.913061    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tfn6f"]
Sep 18 06:35:14.193600 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.193565    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"]
Sep 18 06:35:14.193938 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.193863    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa531efb-8c58-4f80-a567-7205699e11a8" containerName="limitador"
Sep 18 06:35:14.193938 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.193873    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa531efb-8c58-4f80-a567-7205699e11a8" containerName="limitador"
Sep 18 06:35:14.193938 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.193932    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa531efb-8c58-4f80-a567-7205699e11a8" containerName="limitador"
Sep 18 06:35:14.198130 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.198106    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.200821 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.200801    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 06:35:14.200940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.200802    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 06:35:14.201901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.201880    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 06:35:14.202053 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.201931    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-q5hk4\""
Sep 18 06:35:14.204643 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.204620    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"]
Sep 18 06:35:14.291237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.291199    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-data\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.291237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.291233    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4md7\" (UniqueName: \"kubernetes.io/projected/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-kube-api-access-f4md7\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.392643 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.392609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-data\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.392643 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.392642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4md7\" (UniqueName: \"kubernetes.io/projected/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-kube-api-access-f4md7\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.392984 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.392967    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-data\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.402795 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.402764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4md7\" (UniqueName: \"kubernetes.io/projected/1909799c-66c4-4efd-bc3c-8adf9d0b78b0-kube-api-access-f4md7\") pod \"postgres-69fb4594cf-d5pdc\" (UID: \"1909799c-66c4-4efd-bc3c-8adf9d0b78b0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.509244 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.509169    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:14.776836 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.776764    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"]
Sep 18 06:35:14.780365 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:35:14.780329    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1909799c_66c4_4efd_bc3c_8adf9d0b78b0.slice/crio-7c09ee9ec22df40c0c252ef465a119321625865fe5d21111c40d24e22c62e299 WatchSource:0}: Error finding container 7c09ee9ec22df40c0c252ef465a119321625865fe5d21111c40d24e22c62e299: Status 404 returned error can't find the container with id 7c09ee9ec22df40c0c252ef465a119321625865fe5d21111c40d24e22c62e299
Sep 18 06:35:14.910036 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.910001    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa531efb-8c58-4f80-a567-7205699e11a8" path="/var/lib/kubelet/pods/aa531efb-8c58-4f80-a567-7205699e11a8/volumes"
Sep 18 06:35:14.910319 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:14.910298    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc" event={"ID":"1909799c-66c4-4efd-bc3c-8adf9d0b78b0","Type":"ContainerStarted","Data":"7c09ee9ec22df40c0c252ef465a119321625865fe5d21111c40d24e22c62e299"}
Sep 18 06:35:19.931655 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:19.931573    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc" event={"ID":"1909799c-66c4-4efd-bc3c-8adf9d0b78b0","Type":"ContainerStarted","Data":"8817d0aeb1b1b4d8fcba1ed52b9d8165ca37c32eb4720d951ca259da402b0be0"}
Sep 18 06:35:19.931998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:19.931705    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:19.948596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:19.948552    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc" podStartSLOduration=1.24788945 podStartE2EDuration="5.948539833s" totalImagesPullingTime="4.700650383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:35:19.946582033 +0000 UTC m=+799.529967682" watchObservedRunningTime="2026-09-18 06:35:19.948539833 +0000 UTC m=+799.531925485"
Sep 18 06:35:25.965059 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:25.964977    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d5pdc"
Sep 18 06:35:28.763730 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:28.763696    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:35:28.769499 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:28.764313    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-xnz85" podUID="99903ab2-ab74-43ac-929a-3a4e54c32535" containerName="authorino" containerID="cri-o://be65d7dafd9d0119fb6f1c0108dfa3ab972cdd696b3b7e01f14d3d7cd7b3d285" gracePeriod=30
Sep 18 06:35:28.972821 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:28.972731    2567 generic.go:356] "Generic (PLEG): container finished" podID="99903ab2-ab74-43ac-929a-3a4e54c32535" containerID="be65d7dafd9d0119fb6f1c0108dfa3ab972cdd696b3b7e01f14d3d7cd7b3d285" exitCode=0
Sep 18 06:35:28.972821 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:28.972778    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xnz85" event={"ID":"99903ab2-ab74-43ac-929a-3a4e54c32535","Type":"ContainerDied","Data":"be65d7dafd9d0119fb6f1c0108dfa3ab972cdd696b3b7e01f14d3d7cd7b3d285"}
Sep 18 06:35:29.069334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.069306    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:35:29.121694 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.121664    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tcgk\" (UniqueName: \"kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk\") pod \"99903ab2-ab74-43ac-929a-3a4e54c32535\" (UID: \"99903ab2-ab74-43ac-929a-3a4e54c32535\") "
Sep 18 06:35:29.123652 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.123624    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk" (OuterVolumeSpecName: "kube-api-access-5tcgk") pod "99903ab2-ab74-43ac-929a-3a4e54c32535" (UID: "99903ab2-ab74-43ac-929a-3a4e54c32535"). InnerVolumeSpecName "kube-api-access-5tcgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:35:29.223246 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.223215    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tcgk\" (UniqueName: \"kubernetes.io/projected/99903ab2-ab74-43ac-929a-3a4e54c32535-kube-api-access-5tcgk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:29.979512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.979477    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-xnz85" event={"ID":"99903ab2-ab74-43ac-929a-3a4e54c32535","Type":"ContainerDied","Data":"7d0c563bea03dc3e732dd0446f1e04d241df63f5ca9209a8b499f01221d40486"}
Sep 18 06:35:29.979512 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.979508    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-xnz85"
Sep 18 06:35:29.979986 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.979535    2567 scope.go:117] "RemoveContainer" containerID="be65d7dafd9d0119fb6f1c0108dfa3ab972cdd696b3b7e01f14d3d7cd7b3d285"
Sep 18 06:35:29.999076 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:29.999044    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:35:30.002258 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:30.002237    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-xnz85"]
Sep 18 06:35:30.909544 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:30.909507    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99903ab2-ab74-43ac-929a-3a4e54c32535" path="/var/lib/kubelet/pods/99903ab2-ab74-43ac-929a-3a4e54c32535/volumes"
Sep 18 06:35:33.239934 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.239899    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:33.240300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.240253    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99903ab2-ab74-43ac-929a-3a4e54c32535" containerName="authorino"
Sep 18 06:35:33.240300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.240266    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99903ab2-ab74-43ac-929a-3a4e54c32535" containerName="authorino"
Sep 18 06:35:33.240374 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.240331    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99903ab2-ab74-43ac-929a-3a4e54c32535" containerName="authorino"
Sep 18 06:35:33.244680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.244663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.248054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.248027    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 06:35:33.248185 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.248077    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 06:35:33.248185 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.248035    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b28rl\""
Sep 18 06:35:33.248185 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.248040    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 06:35:33.252740 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.252720    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:33.364244 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.364210    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.364433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.364255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.364433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.364318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgfx\" (UniqueName: \"kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.410914 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.410883    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 06:35:33.413345 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.413329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.422565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.422540    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 06:35:33.465470 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.465445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.465598 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.465483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.465598 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.465510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgfx\" (UniqueName: \"kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.465709 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:35:33.465604    2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 18 06:35:33.465709 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:35:33.465685    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert podName:8a424940-61b2-4315-b1a7-d906f576cf6d nodeName:}" failed. No retries permitted until 2026-09-18 06:35:33.965670694 +0000 UTC m=+813.549056328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert") pod "maas-controller-f7c9fdfb-fnvfk" (UID: "8a424940-61b2-4315-b1a7-d906f576cf6d") : secret "maas-controller-webhook-cert" not found
Sep 18 06:35:33.467850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.467826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.476027 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.476005    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgfx\" (UniqueName: \"kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.566827 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.566748    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfb7\" (UniqueName: \"kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.566827 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.566791    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.566827 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.566806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.667609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.667575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.667609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.667608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.667798 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.667686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfb7\" (UniqueName: \"kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.670041 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.670006    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.670174 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.670072    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.675767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.675747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfb7\" (UniqueName: \"kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7\") pod \"maas-controller-84c8b64bc5-v9dlb\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") " pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.724296 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.724266    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:33.970297 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.970264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.972675 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.972652    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") pod \"maas-controller-f7c9fdfb-fnvfk\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") " pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:33.995636 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:33.993717    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 06:35:34.002681 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:34.002649    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" event={"ID":"555fa4e1-7fbb-4a94-b788-cfa495d55553","Type":"ContainerStarted","Data":"824eeca29c4c55e89a560fa83c75a1b3031a68e492332a4eb853282442a5963a"}
Sep 18 06:35:34.155368 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:34.155334    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:34.422784 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:34.422757    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:34.425246 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:35:34.425222    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a424940_61b2_4315_b1a7_d906f576cf6d.slice/crio-a4da31f92b1af8e837f9ec76d480f2eee388b567234e59bc25d50ad888217207 WatchSource:0}: Error finding container a4da31f92b1af8e837f9ec76d480f2eee388b567234e59bc25d50ad888217207: Status 404 returned error can't find the container with id a4da31f92b1af8e837f9ec76d480f2eee388b567234e59bc25d50ad888217207
Sep 18 06:35:35.011078 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:35.011023    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" event={"ID":"8a424940-61b2-4315-b1a7-d906f576cf6d","Type":"ContainerStarted","Data":"a4da31f92b1af8e837f9ec76d480f2eee388b567234e59bc25d50ad888217207"}
Sep 18 06:35:37.022567 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.022508    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" event={"ID":"555fa4e1-7fbb-4a94-b788-cfa495d55553","Type":"ContainerStarted","Data":"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"}
Sep 18 06:35:37.023121 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.022712    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:37.024433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.024404    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" event={"ID":"8a424940-61b2-4315-b1a7-d906f576cf6d","Type":"ContainerStarted","Data":"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"}
Sep 18 06:35:37.024556 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.024532    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:37.040003 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.039961    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" podStartSLOduration=1.47015337 podStartE2EDuration="4.039947684s" totalImagesPullingTime="2.569794314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:35:37.038802095 +0000 UTC m=+816.622187762" watchObservedRunningTime="2026-09-18 06:35:37.039947684 +0000 UTC m=+816.623333337"
Sep 18 06:35:37.056060 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:37.056016    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" podStartSLOduration=1.904365069 podStartE2EDuration="4.056002647s" totalImagesPullingTime="2.151637578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:35:37.055095407 +0000 UTC m=+816.638481060" watchObservedRunningTime="2026-09-18 06:35:37.056002647 +0000 UTC m=+816.639388315"
Sep 18 06:35:48.032109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.032074    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:48.032490 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.032295    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 06:35:48.086054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.086022    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:48.086257 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.086234    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" podUID="8a424940-61b2-4315-b1a7-d906f576cf6d" containerName="manager" containerID="cri-o://eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573" gracePeriod=30
Sep 18 06:35:48.328608 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.328581    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:48.502774 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.502731    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftgfx\" (UniqueName: \"kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx\") pod \"8a424940-61b2-4315-b1a7-d906f576cf6d\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") "
Sep 18 06:35:48.502774 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.502789    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs\") pod \"8a424940-61b2-4315-b1a7-d906f576cf6d\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") "
Sep 18 06:35:48.502985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.502892    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") pod \"8a424940-61b2-4315-b1a7-d906f576cf6d\" (UID: \"8a424940-61b2-4315-b1a7-d906f576cf6d\") "
Sep 18 06:35:48.504965 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.504939    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a424940-61b2-4315-b1a7-d906f576cf6d" (UID: "8a424940-61b2-4315-b1a7-d906f576cf6d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:35:48.505084 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.505033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a424940-61b2-4315-b1a7-d906f576cf6d" (UID: "8a424940-61b2-4315-b1a7-d906f576cf6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:35:48.505084 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.505063    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx" (OuterVolumeSpecName: "kube-api-access-ftgfx") pod "8a424940-61b2-4315-b1a7-d906f576cf6d" (UID: "8a424940-61b2-4315-b1a7-d906f576cf6d"). InnerVolumeSpecName "kube-api-access-ftgfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:35:48.603812 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.603731    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgfx\" (UniqueName: \"kubernetes.io/projected/8a424940-61b2-4315-b1a7-d906f576cf6d-kube-api-access-ftgfx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:48.603812 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.603758    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:48.603812 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:48.603768    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a424940-61b2-4315-b1a7-d906f576cf6d-webhook-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:35:49.078873 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.078839    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a424940-61b2-4315-b1a7-d906f576cf6d" containerID="eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573" exitCode=0
Sep 18 06:35:49.079292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.078886    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk"
Sep 18 06:35:49.079292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.078930    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" event={"ID":"8a424940-61b2-4315-b1a7-d906f576cf6d","Type":"ContainerDied","Data":"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"}
Sep 18 06:35:49.079292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.078982    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f7c9fdfb-fnvfk" event={"ID":"8a424940-61b2-4315-b1a7-d906f576cf6d","Type":"ContainerDied","Data":"a4da31f92b1af8e837f9ec76d480f2eee388b567234e59bc25d50ad888217207"}
Sep 18 06:35:49.079292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.079005    2567 scope.go:117] "RemoveContainer" containerID="eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"
Sep 18 06:35:49.087423 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.087237    2567 scope.go:117] "RemoveContainer" containerID="eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"
Sep 18 06:35:49.087521 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:35:49.087503    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573\": container with ID starting with eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573 not found: ID does not exist" containerID="eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"
Sep 18 06:35:49.087564 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.087530    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573"} err="failed to get container status \"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573\": rpc error: code = NotFound desc = could not find container \"eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573\": container with ID starting with eba560b3b350425d3de257daa171d969f62a3fe49a08746fef2236fac7818573 not found: ID does not exist"
Sep 18 06:35:49.095447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.095424    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:49.101846 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:49.101822    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f7c9fdfb-fnvfk"]
Sep 18 06:35:50.908587 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:35:50.908554    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a424940-61b2-4315-b1a7-d906f576cf6d" path="/var/lib/kubelet/pods/8a424940-61b2-4315-b1a7-d906f576cf6d/volumes"
Sep 18 06:36:09.824190 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.824129    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 06:36:09.824693 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.824640    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a424940-61b2-4315-b1a7-d906f576cf6d" containerName="manager"
Sep 18 06:36:09.824693 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.824660    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a424940-61b2-4315-b1a7-d906f576cf6d" containerName="manager"
Sep 18 06:36:09.824814 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.824772    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a424940-61b2-4315-b1a7-d906f576cf6d" containerName="manager"
Sep 18 06:36:09.829203 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.829180    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.832619 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.832595    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 06:36:09.832747 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.832659    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 06:36:09.832747 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.832604    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mxmzc\""
Sep 18 06:36:09.838855 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.838831    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 06:36:09.871374 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.871343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.871518 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.871377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gln\" (UniqueName: \"kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.871518 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.871440    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.871518 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.871472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.972039 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.972002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.972039 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.972039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gln\" (UniqueName: \"kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.972314 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.972083    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.972314 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.972114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.972314 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:36:09.972240    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 18 06:36:09.972314 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:36:09.972309    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs podName:2a2e3b52-2a18-44e4-ad13-82c2e0d525e8 nodeName:}" failed. No retries permitted until 2026-09-18 06:36:10.472291446 +0000 UTC m=+850.055677094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs") pod "maas-api-dc699bbb9-fbxh9" (UID: "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8") : secret "maas-api-metrics-tls" not found
Sep 18 06:36:09.972694 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.972676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.974730 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.974704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:09.980515 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:09.980493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gln\" (UniqueName: \"kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:10.396255 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.396216    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"]
Sep 18 06:36:10.398713 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.398691    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.401199 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.401177    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kv4r\""
Sep 18 06:36:10.401301 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.401257    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 06:36:10.410980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.410957    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"]
Sep 18 06:36:10.475534 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.475506    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:10.475656 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.475553    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxlr\" (UniqueName: \"kubernetes.io/projected/d7b972cc-f0d8-4691-956c-89dd94200ae8-kube-api-access-7jxlr\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.475698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.475670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b972cc-f0d8-4691-956c-89dd94200ae8-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.477940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.477921    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") pod \"maas-api-dc699bbb9-fbxh9\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") " pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:10.577150 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.577111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b972cc-f0d8-4691-956c-89dd94200ae8-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.577345 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.577255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxlr\" (UniqueName: \"kubernetes.io/projected/d7b972cc-f0d8-4691-956c-89dd94200ae8-kube-api-access-7jxlr\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.577825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.577801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7b972cc-f0d8-4691-956c-89dd94200ae8-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.586028 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.586004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxlr\" (UniqueName: \"kubernetes.io/projected/d7b972cc-f0d8-4691-956c-89dd94200ae8-kube-api-access-7jxlr\") pod \"payload-pre-processing-59fbc47bf8-4s9v7\" (UID: \"d7b972cc-f0d8-4691-956c-89dd94200ae8\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.709316 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.709285    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:10.739557 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.739514    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:10.976835 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:10.976808    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"]
Sep 18 06:36:10.978654 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:36:10.978629    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b972cc_f0d8_4691_956c_89dd94200ae8.slice/crio-d631f97f3c91cc3c21f0633f9b04f28782e656d1b94cc4a201a130a66d3edc33 WatchSource:0}: Error finding container d631f97f3c91cc3c21f0633f9b04f28782e656d1b94cc4a201a130a66d3edc33: Status 404 returned error can't find the container with id d631f97f3c91cc3c21f0633f9b04f28782e656d1b94cc4a201a130a66d3edc33
Sep 18 06:36:11.015375 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:11.015355    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 06:36:11.017805 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:36:11.017771    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2e3b52_2a18_44e4_ad13_82c2e0d525e8.slice/crio-9c5036183827387378c224bc0514352da6b1537d8011b8f51fb39edd90130529 WatchSource:0}: Error finding container 9c5036183827387378c224bc0514352da6b1537d8011b8f51fb39edd90130529: Status 404 returned error can't find the container with id 9c5036183827387378c224bc0514352da6b1537d8011b8f51fb39edd90130529
Sep 18 06:36:11.181660 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:11.181616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" event={"ID":"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8","Type":"ContainerStarted","Data":"9c5036183827387378c224bc0514352da6b1537d8011b8f51fb39edd90130529"}
Sep 18 06:36:11.183002 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:11.182975    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7" event={"ID":"d7b972cc-f0d8-4691-956c-89dd94200ae8","Type":"ContainerStarted","Data":"d631f97f3c91cc3c21f0633f9b04f28782e656d1b94cc4a201a130a66d3edc33"}
Sep 18 06:36:14.202185 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.202130    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" event={"ID":"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8","Type":"ContainerStarted","Data":"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"}
Sep 18 06:36:14.202605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.202242    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:14.203759 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.203730    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7" event={"ID":"d7b972cc-f0d8-4691-956c-89dd94200ae8","Type":"ContainerStarted","Data":"6f7b3d07a7b386efc0401f6a284b34bdde9d23efcd9c6ba910db545ea9405629"}
Sep 18 06:36:14.203867 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.203790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:36:14.217749 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.217708    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" podStartSLOduration=2.410219234 podStartE2EDuration="5.217696259s" totalImagesPullingTime="2.807477025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:36:14.217474257 +0000 UTC m=+853.800859911" watchObservedRunningTime="2026-09-18 06:36:14.217696259 +0000 UTC m=+853.801081912"
Sep 18 06:36:14.236022 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:14.235975    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7" podStartSLOduration=1.3880061719999999 podStartE2EDuration="4.235961128s" totalImagesPullingTime="2.847954956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:36:14.235249627 +0000 UTC m=+853.818635284" watchObservedRunningTime="2026-09-18 06:36:14.235961128 +0000 UTC m=+853.819346780"
Sep 18 06:36:20.213236 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:20.213205    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 06:36:25.210628 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:36:25.210597    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4s9v7"
Sep 18 06:37:00.859932 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:00.859899    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:37:00.860995 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:00.860973    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:37:10.694049 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.694010    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"]
Sep 18 06:37:10.697518 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.697497    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.700044 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.700022    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 06:37:10.700173 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.700065    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wzjhw\""
Sep 18 06:37:10.700173 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.700115    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 06:37:10.701320 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.701300    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 06:37:10.712854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.712825    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"]
Sep 18 06:37:10.798697 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.798872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7h92\" (UniqueName: \"kubernetes.io/projected/c97cf146-3055-4b25-a420-56320347d03c-kube-api-access-n7h92\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.798872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798780    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97cf146-3055-4b25-a420-56320347d03c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.798987 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.798987 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.798987 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.798936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899689 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7h92\" (UniqueName: \"kubernetes.io/projected/c97cf146-3055-4b25-a420-56320347d03c-kube-api-access-n7h92\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97cf146-3055-4b25-a420-56320347d03c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899752    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.899871 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.899800    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.900129 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.900108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.900209 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.900186    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.900301 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.900275    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.902089 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.902070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97cf146-3055-4b25-a420-56320347d03c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.902511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.902490    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97cf146-3055-4b25-a420-56320347d03c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:10.906810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:10.906788    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7h92\" (UniqueName: \"kubernetes.io/projected/c97cf146-3055-4b25-a420-56320347d03c-kube-api-access-n7h92\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jxczh\" (UID: \"c97cf146-3055-4b25-a420-56320347d03c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:11.014354 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:11.014275    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:11.287830 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:11.287741    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"]
Sep 18 06:37:11.290906 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:37:11.290875    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97cf146_3055_4b25_a420_56320347d03c.slice/crio-f70cd68c2fd95cac712d27a38270e7e3b5ebe1198f142bc24767af5daa7b21fb WatchSource:0}: Error finding container f70cd68c2fd95cac712d27a38270e7e3b5ebe1198f142bc24767af5daa7b21fb: Status 404 returned error can't find the container with id f70cd68c2fd95cac712d27a38270e7e3b5ebe1198f142bc24767af5daa7b21fb
Sep 18 06:37:11.292242 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:11.292223    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:37:11.459565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:11.459523    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh" event={"ID":"c97cf146-3055-4b25-a420-56320347d03c","Type":"ContainerStarted","Data":"f70cd68c2fd95cac712d27a38270e7e3b5ebe1198f142bc24767af5daa7b21fb"}
Sep 18 06:37:12.465714 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:12.465677    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh" event={"ID":"c97cf146-3055-4b25-a420-56320347d03c","Type":"ContainerStarted","Data":"190248a055052dba354e2c6617057cd8019244ba3ffd780ead88eed47d1499a4"}
Sep 18 06:37:12.483014 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:12.482965    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh" podStartSLOduration=1.476958501 podStartE2EDuration="2.482950394s" totalImagesPullingTime="1.005991893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:37:12.48130447 +0000 UTC m=+912.064690124" watchObservedRunningTime="2026-09-18 06:37:12.482950394 +0000 UTC m=+912.066336046"
Sep 18 06:37:21.014556 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:21.014523    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:21.014975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:21.014574    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:21.026971 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:21.026950    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:21.516114 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:21.516082    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jxczh"
Sep 18 06:37:30.181072 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.181040    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"]
Sep 18 06:37:30.184329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.184308    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.186617 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.186596    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 06:37:30.194035 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.194011    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"]
Sep 18 06:37:30.362939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.362904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.363109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.362958    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.363109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.362984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.363109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.363030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.363109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.363045    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a966e4b-de55-4d0c-9b54-7a49613f5341-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.363109 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.363076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8n7\" (UniqueName: \"kubernetes.io/projected/7a966e4b-de55-4d0c-9b54-7a49613f5341-kube-api-access-lv8n7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464488 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a966e4b-de55-4d0c-9b54-7a49613f5341-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8n7\" (UniqueName: \"kubernetes.io/projected/7a966e4b-de55-4d0c-9b54-7a49613f5341-kube-api-access-lv8n7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.464894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464844    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.465042 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.464934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.465160 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.465117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.466852 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.466826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a966e4b-de55-4d0c-9b54-7a49613f5341-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.467284 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.467261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a966e4b-de55-4d0c-9b54-7a49613f5341-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.472151 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.472115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8n7\" (UniqueName: \"kubernetes.io/projected/7a966e4b-de55-4d0c-9b54-7a49613f5341-kube-api-access-lv8n7\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7\" (UID: \"7a966e4b-de55-4d0c-9b54-7a49613f5341\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.495740 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.495719    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:30.763797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:30.763766    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"]
Sep 18 06:37:30.767503 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:37:30.767475    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a966e4b_de55_4d0c_9b54_7a49613f5341.slice/crio-a7c4216bf06a466754eaadd401757a596531f8d169e1f7c1a6f314d19468b0d0 WatchSource:0}: Error finding container a7c4216bf06a466754eaadd401757a596531f8d169e1f7c1a6f314d19468b0d0: Status 404 returned error can't find the container with id a7c4216bf06a466754eaadd401757a596531f8d169e1f7c1a6f314d19468b0d0
Sep 18 06:37:31.551703 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:31.551658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7" event={"ID":"7a966e4b-de55-4d0c-9b54-7a49613f5341","Type":"ContainerStarted","Data":"e4c2b48202c43996dd4bd5ec3b131de4a05e99e06dbd3d64e6b2094aa0f27fbc"}
Sep 18 06:37:31.552071 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:31.551723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7" event={"ID":"7a966e4b-de55-4d0c-9b54-7a49613f5341","Type":"ContainerStarted","Data":"a7c4216bf06a466754eaadd401757a596531f8d169e1f7c1a6f314d19468b0d0"}
Sep 18 06:37:31.570033 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:31.569985    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7" podStartSLOduration=1.391129898 podStartE2EDuration="1.569969697s" totalImagesPullingTime="178.839799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:37:31.568053051 +0000 UTC m=+931.151438701" watchObservedRunningTime="2026-09-18 06:37:31.569969697 +0000 UTC m=+931.153355387"
Sep 18 06:37:40.496911 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:40.496875    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:40.497442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:40.496929    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:40.509334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:40.509312    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:37:40.602550 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:37:40.602518    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7"
Sep 18 06:40:04.291093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.291003    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b64bb474f-h7xxg"]
Sep 18 06:40:04.294335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.294312    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.297675 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.297652    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zcv2z\""
Sep 18 06:40:04.297798 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.297652    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 06:40:04.302114 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.302088    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b64bb474f-h7xxg"]
Sep 18 06:40:04.361768 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.361726    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/922690f5-a238-4ad3-84fc-660fcab998e6-tls-cert\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.361946 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.361787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7ln\" (UniqueName: \"kubernetes.io/projected/922690f5-a238-4ad3-84fc-660fcab998e6-kube-api-access-qn7ln\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.462696 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.462661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/922690f5-a238-4ad3-84fc-660fcab998e6-tls-cert\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.462854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.462718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7ln\" (UniqueName: \"kubernetes.io/projected/922690f5-a238-4ad3-84fc-660fcab998e6-kube-api-access-qn7ln\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.465175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.465153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/922690f5-a238-4ad3-84fc-660fcab998e6-tls-cert\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.469988 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.469959    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7ln\" (UniqueName: \"kubernetes.io/projected/922690f5-a238-4ad3-84fc-660fcab998e6-kube-api-access-qn7ln\") pod \"authorino-7b64bb474f-h7xxg\" (UID: \"922690f5-a238-4ad3-84fc-660fcab998e6\") " pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.603346 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.603256    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b64bb474f-h7xxg"
Sep 18 06:40:04.871087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:04.871030    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b64bb474f-h7xxg"]
Sep 18 06:40:04.873950 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:04.873925    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod922690f5_a238_4ad3_84fc_660fcab998e6.slice/crio-14027f308f86d84c46bad13a37208a71c6c9a146bcf435e0ccd7a4b96d8856f6 WatchSource:0}: Error finding container 14027f308f86d84c46bad13a37208a71c6c9a146bcf435e0ccd7a4b96d8856f6: Status 404 returned error can't find the container with id 14027f308f86d84c46bad13a37208a71c6c9a146bcf435e0ccd7a4b96d8856f6
Sep 18 06:40:05.233475 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:05.233427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b64bb474f-h7xxg" event={"ID":"922690f5-a238-4ad3-84fc-660fcab998e6","Type":"ContainerStarted","Data":"14027f308f86d84c46bad13a37208a71c6c9a146bcf435e0ccd7a4b96d8856f6"}
Sep 18 06:40:06.240485 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:06.240441    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b64bb474f-h7xxg" event={"ID":"922690f5-a238-4ad3-84fc-660fcab998e6","Type":"ContainerStarted","Data":"f5da9586592cf4692d1acfc02467a8bf18241ec36a6d4e6c4a7e24f41b2e89b3"}
Sep 18 06:40:06.254231 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:06.254179    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b64bb474f-h7xxg" podStartSLOduration=1.641606331 podStartE2EDuration="2.254163792s" totalImagesPullingTime="612.557461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:06.252707943 +0000 UTC m=+1085.836093596" watchObservedRunningTime="2026-09-18 06:40:06.254163792 +0000 UTC m=+1085.837549482"
Sep 18 06:40:27.944085 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.944049    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:40:27.948095 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.948074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:27.950852 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.950831    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-654463-serving-cert\""
Sep 18 06:40:27.959478 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.959454    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:40:27.968276 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.968242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mql9t\" (UniqueName: \"kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:27.968384 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.968282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:27.968435 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.968403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:27.968483 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:27.968461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.069261 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.069226    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.069449 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.069392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mql9t\" (UniqueName: \"kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.069449 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.069435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.069565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.069491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.069898 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.069867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.071907 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.071884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.072103 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.072085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.078279 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.078259    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mql9t\" (UniqueName: \"kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t\") pod \"maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.259952 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.259864    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:28.536458 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.536428    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:40:28.538044 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:28.538004    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863c0a4a_05ce_46dd_b947_c01c9641e9cb.slice/crio-e49f1bfe6be3b31e7556b89cbdf7ff012391e6bc7b9cae53c041ca89b2013841 WatchSource:0}: Error finding container e49f1bfe6be3b31e7556b89cbdf7ff012391e6bc7b9cae53c041ca89b2013841: Status 404 returned error can't find the container with id e49f1bfe6be3b31e7556b89cbdf7ff012391e6bc7b9cae53c041ca89b2013841
Sep 18 06:40:28.671616 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.671582    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:40:28.675266 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.675242    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.677765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.677741    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-654463\""
Sep 18 06:40:28.677861 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.677786    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-654463-dockercfg-jddxm\""
Sep 18 06:40:28.686437 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.686414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:40:28.777042 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.777011    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.777042 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.777045    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd56z\" (UniqueName: \"kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.877738 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.877665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.877738 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.877700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd56z\" (UniqueName: \"kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.878397 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.878378    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.885085 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.885060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd56z\" (UniqueName: \"kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z\") pod \"payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:28.910857 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.910833    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:40:28.914300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.914279    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:28.916758 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.916733    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7c4f4b-serving-cert\""
Sep 18 06:40:28.923700 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.923456    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:40:28.978991 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.978956    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:28.979439 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.979012    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:28.979439 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.979031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmtf\" (UniqueName: \"kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:28.979439 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.979067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:28.986889 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:28.986865    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:29.080269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.080245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.080401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.080282    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.080401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.080301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmtf\" (UniqueName: \"kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.080401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.080329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.080558 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:29.080455    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-serving-cert: secret "maas-api-e2e-inf-a-7c4f4b-serving-cert" not found
Sep 18 06:40:29.080558 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:29.080500    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls podName:c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1 nodeName:}" failed. No retries permitted until 2026-09-18 06:40:29.580484555 +0000 UTC m=+1109.163870186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls") pod "maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" (UID: "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1") : secret "maas-api-e2e-inf-a-7c4f4b-serving-cert" not found
Sep 18 06:40:29.080974 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.080954    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.082641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.082617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.088983 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.088958    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmtf\" (UniqueName: \"kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.262585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.262558    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:40:29.264844 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:29.264817    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd833dc52_82f6_465e_b59a_412709b25164.slice/crio-559ae90f75476abab9149ef6bb0a0620c0c9486eebdf0472d1172a30bc60086e WatchSource:0}: Error finding container 559ae90f75476abab9149ef6bb0a0620c0c9486eebdf0472d1172a30bc60086e: Status 404 returned error can't find the container with id 559ae90f75476abab9149ef6bb0a0620c0c9486eebdf0472d1172a30bc60086e
Sep 18 06:40:29.353329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.353292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" event={"ID":"863c0a4a-05ce-46dd-b947-c01c9641e9cb","Type":"ContainerStarted","Data":"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"}
Sep 18 06:40:29.353565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.353546    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:29.353718 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.353700    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" event={"ID":"863c0a4a-05ce-46dd-b947-c01c9641e9cb","Type":"ContainerStarted","Data":"e49f1bfe6be3b31e7556b89cbdf7ff012391e6bc7b9cae53c041ca89b2013841"}
Sep 18 06:40:29.355002 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.354977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" event={"ID":"d833dc52-82f6-465e-b59a-412709b25164","Type":"ContainerStarted","Data":"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"}
Sep 18 06:40:29.355108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.355012    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" event={"ID":"d833dc52-82f6-465e-b59a-412709b25164","Type":"ContainerStarted","Data":"559ae90f75476abab9149ef6bb0a0620c0c9486eebdf0472d1172a30bc60086e"}
Sep 18 06:40:29.355181 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.355158    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:29.373321 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.373268    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" podStartSLOduration=1.8738526819999999 podStartE2EDuration="2.373247561s" totalImagesPullingTime="499.394879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:29.370978119 +0000 UTC m=+1108.954363775" watchObservedRunningTime="2026-09-18 06:40:29.373247561 +0000 UTC m=+1108.956633214"
Sep 18 06:40:29.388789 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.388662    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" podStartSLOduration=1.388642599 podStartE2EDuration="1.388642599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:29.387745752 +0000 UTC m=+1108.971131406" watchObservedRunningTime="2026-09-18 06:40:29.388642599 +0000 UTC m=+1108.972028255"
Sep 18 06:40:29.496604 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.496569    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:40:29.500292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.500270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.502569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.502545    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7c4f4b\""
Sep 18 06:40:29.510108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.510081    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:40:29.534770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.534739    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:40:29.538371 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.538350    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.540811 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.540789    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7c4f4b-dockercfg-trkjr\""
Sep 18 06:40:29.548367 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.548343    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:40:29.585601 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.585564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.585770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.585628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.585770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.585677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.585770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.585705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvwl\" (UniqueName: \"kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.585770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.585729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjd5\" (UniqueName: \"kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.588205 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.588180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") pod \"maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.687093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.687004    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.687093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.687066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvwl\" (UniqueName: \"kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.687093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.687090    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjd5\" (UniqueName: \"kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.687385 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.687227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.687945 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.687913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.688033 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.688010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.695198 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.695176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjd5\" (UniqueName: \"kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5\") pod \"payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:29.695289 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.695248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvwl\" (UniqueName: \"kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl\") pod \"payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.813547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.813510    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:29.826279 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.826258    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:29.853250 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:29.853219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:30.096548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.096421    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:40:30.099513 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:30.099486    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115b42e3_d1bd_4608_8fcb_abb663825392.slice/crio-9ade35df28f37073bd0bc0063a1e273a8f693c4ab5e69eb306abab888d6fc493 WatchSource:0}: Error finding container 9ade35df28f37073bd0bc0063a1e273a8f693c4ab5e69eb306abab888d6fc493: Status 404 returned error can't find the container with id 9ade35df28f37073bd0bc0063a1e273a8f693c4ab5e69eb306abab888d6fc493
Sep 18 06:40:30.115174 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.115127    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:40:30.117471 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:30.117437    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f8dcab_7b5b_45bf_871c_4a9036fd7ea1.slice/crio-6b39e3b48f7b6022ee6d3a6c73049c4e82ba1efecaf85358877b776bad05e8de WatchSource:0}: Error finding container 6b39e3b48f7b6022ee6d3a6c73049c4e82ba1efecaf85358877b776bad05e8de: Status 404 returned error can't find the container with id 6b39e3b48f7b6022ee6d3a6c73049c4e82ba1efecaf85358877b776bad05e8de
Sep 18 06:40:30.146338 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.146312    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:40:30.148197 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:30.148169    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13114a20_a24d_4576_b8b9_718db6509f4e.slice/crio-aa700ca974b6b15018a1daa9e2b546861a9a3f27adfd4273c25ba44e07d2a634 WatchSource:0}: Error finding container aa700ca974b6b15018a1daa9e2b546861a9a3f27adfd4273c25ba44e07d2a634: Status 404 returned error can't find the container with id aa700ca974b6b15018a1daa9e2b546861a9a3f27adfd4273c25ba44e07d2a634
Sep 18 06:40:30.364242 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.363850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" event={"ID":"115b42e3-d1bd-4608-8fcb-abb663825392","Type":"ContainerStarted","Data":"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"}
Sep 18 06:40:30.364242 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.363919    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" event={"ID":"115b42e3-d1bd-4608-8fcb-abb663825392","Type":"ContainerStarted","Data":"9ade35df28f37073bd0bc0063a1e273a8f693c4ab5e69eb306abab888d6fc493"}
Sep 18 06:40:30.364242 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.363982    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:30.366207 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.366179    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" event={"ID":"13114a20-a24d-4576-b8b9-718db6509f4e","Type":"ContainerStarted","Data":"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"}
Sep 18 06:40:30.366351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.366230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" event={"ID":"13114a20-a24d-4576-b8b9-718db6509f4e","Type":"ContainerStarted","Data":"aa700ca974b6b15018a1daa9e2b546861a9a3f27adfd4273c25ba44e07d2a634"}
Sep 18 06:40:30.366351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.366259    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:30.367732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.367689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" event={"ID":"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1","Type":"ContainerStarted","Data":"6b39e3b48f7b6022ee6d3a6c73049c4e82ba1efecaf85358877b776bad05e8de"}
Sep 18 06:40:30.379626 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.379581    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" podStartSLOduration=1.379567663 podStartE2EDuration="1.379567663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:30.378374494 +0000 UTC m=+1109.961760149" watchObservedRunningTime="2026-09-18 06:40:30.379567663 +0000 UTC m=+1109.962953315"
Sep 18 06:40:30.394073 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:30.394021    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" podStartSLOduration=1.3940083300000001 podStartE2EDuration="1.39400833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:30.393381247 +0000 UTC m=+1109.976766902" watchObservedRunningTime="2026-09-18 06:40:30.39400833 +0000 UTC m=+1109.977393983"
Sep 18 06:40:31.375563 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:31.375528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" event={"ID":"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1","Type":"ContainerStarted","Data":"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"}
Sep 18 06:40:31.376052 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:31.375974    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:31.393999 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:31.393941    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" podStartSLOduration=3.115514701 podStartE2EDuration="3.39392698s" totalImagesPullingTime="278.412279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:31.392155326 +0000 UTC m=+1110.975540977" watchObservedRunningTime="2026-09-18 06:40:31.39392698 +0000 UTC m=+1110.977312669"
Sep 18 06:40:35.374622 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:35.374592    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:40:38.393746 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.393713    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:40:38.397843 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.397821    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:40:38.398054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.398032    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.400287 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.400264    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0da33e9c-serving-cert\""
Sep 18 06:40:38.407654 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.407632    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:40:38.476417 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.476384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.476592 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.476571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.476856 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.476820    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxch9\" (UniqueName: \"kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.476946 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.476863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.577368 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.577330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.577540 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.577418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxch9\" (UniqueName: \"kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.577540 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.577438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.577540 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.577464    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.577728 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:38.577557    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-serving-cert: secret "maas-api-e2e-ait-0da33e9c-serving-cert" not found
Sep 18 06:40:38.577728 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:38.577612    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls podName:af83dfd5-f6ce-4faa-a287-4efc9afc167e nodeName:}" failed. No retries permitted until 2026-09-18 06:40:39.077595848 +0000 UTC m=+1118.660981480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls") pod "maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" (UID: "af83dfd5-f6ce-4faa-a287-4efc9afc167e") : secret "maas-api-e2e-ait-0da33e9c-serving-cert" not found
Sep 18 06:40:38.578106 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.578081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.579884 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.579857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:38.585383 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:38.585367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxch9\" (UniqueName: \"kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:39.082985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:39.082944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:39.085596 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:39.085559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") pod \"maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:39.313291 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:39.313257    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:39.585576 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:39.585551    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:40:39.588029 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:39.587997    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf83dfd5_f6ce_4faa_a287_4efc9afc167e.slice/crio-92f087bebc819414d11a0e8815398629c479e81ff6ea8101b3b64039db67479e WatchSource:0}: Error finding container 92f087bebc819414d11a0e8815398629c479e81ff6ea8101b3b64039db67479e: Status 404 returned error can't find the container with id 92f087bebc819414d11a0e8815398629c479e81ff6ea8101b3b64039db67479e
Sep 18 06:40:39.997781 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:39.997742    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:40:40.001380 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.001358    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.003556 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.003536    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-43fdfe-serving-cert\""
Sep 18 06:40:40.012678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.012580    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:40:40.094202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.094172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.094401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.094212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt4sk\" (UniqueName: \"kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.094401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.094324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.094401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.094382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.194940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.194904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.195096 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.194994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.195096 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.195015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt4sk\" (UniqueName: \"kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.195096 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.195040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.195277 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:40.195120    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-serving-cert: secret "maas-api-e2e-autores-43fdfe-serving-cert" not found
Sep 18 06:40:40.195277 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:40:40.195209    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls podName:fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0 nodeName:}" failed. No retries permitted until 2026-09-18 06:40:40.695192774 +0000 UTC m=+1120.278578410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls") pod "maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" (UID: "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0") : secret "maas-api-e2e-autores-43fdfe-serving-cert" not found
Sep 18 06:40:40.195725 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.195703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.197541 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.197515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.203200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.203179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt4sk\" (UniqueName: \"kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.370592 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.370560    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:40:40.426055 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.426017    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" event={"ID":"af83dfd5-f6ce-4faa-a287-4efc9afc167e","Type":"ContainerStarted","Data":"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"}
Sep 18 06:40:40.426270 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.426166    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:40.426270 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.426205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" event={"ID":"af83dfd5-f6ce-4faa-a287-4efc9afc167e","Type":"ContainerStarted","Data":"92f087bebc819414d11a0e8815398629c479e81ff6ea8101b3b64039db67479e"}
Sep 18 06:40:40.444910 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.444861    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" podStartSLOduration=2.134615084 podStartE2EDuration="2.444847247s" totalImagesPullingTime="310.232163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:40.443124155 +0000 UTC m=+1120.026509810" watchObservedRunningTime="2026-09-18 06:40:40.444847247 +0000 UTC m=+1120.028232899"
Sep 18 06:40:40.684723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.684646    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:40:40.688469 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.688447    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.690985 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.690955    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-43fdfe\""
Sep 18 06:40:40.691169 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.691129    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-43fdfe-dockercfg-b8lhc\""
Sep 18 06:40:40.696851 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.696828    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:40:40.699266 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.699243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.701876 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.701847    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") pod \"maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:40.800125 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.800092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rn7\" (UniqueName: \"kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.800285 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.800178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.901581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.901548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rn7\" (UniqueName: \"kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.901733 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.901621    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.902409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.902380    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.909219 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.909124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rn7\" (UniqueName: \"kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7\") pod \"payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:40.914249 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:40.914226    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:41.001110 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.001080    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:41.202527 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.202502    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:40:41.203754 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:41.203723    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3cdb76_8b5b_4e0d_a05d_7084b58e17c0.slice/crio-856786d2b2c0f4bf392ae124af0d75487eede0feb90bf1ce4ecfab0b8ba482f4 WatchSource:0}: Error finding container 856786d2b2c0f4bf392ae124af0d75487eede0feb90bf1ce4ecfab0b8ba482f4: Status 404 returned error can't find the container with id 856786d2b2c0f4bf392ae124af0d75487eede0feb90bf1ce4ecfab0b8ba482f4
Sep 18 06:40:41.275838 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.275809    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:40:41.276783 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:41.276755    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d336b1_9ce2_4293_adf7_9f8e2ffead8c.slice/crio-30f324968f1156e6e8e2e8ed266d093e70540fae9645ef43552e29c4d84d8410 WatchSource:0}: Error finding container 30f324968f1156e6e8e2e8ed266d093e70540fae9645ef43552e29c4d84d8410: Status 404 returned error can't find the container with id 30f324968f1156e6e8e2e8ed266d093e70540fae9645ef43552e29c4d84d8410
Sep 18 06:40:41.377572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.377544    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:40:41.377722 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.377619    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:40:41.432816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.432784    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" event={"ID":"68d336b1-9ce2-4293-adf7-9f8e2ffead8c","Type":"ContainerStarted","Data":"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"}
Sep 18 06:40:41.432989 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.432845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" event={"ID":"68d336b1-9ce2-4293-adf7-9f8e2ffead8c","Type":"ContainerStarted","Data":"30f324968f1156e6e8e2e8ed266d093e70540fae9645ef43552e29c4d84d8410"}
Sep 18 06:40:41.433070 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.432994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:41.434458 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.434428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" event={"ID":"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0","Type":"ContainerStarted","Data":"856786d2b2c0f4bf392ae124af0d75487eede0feb90bf1ce4ecfab0b8ba482f4"}
Sep 18 06:40:41.450747 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:41.450693    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" podStartSLOduration=1.450674567 podStartE2EDuration="1.450674567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:41.448784255 +0000 UTC m=+1121.032169909" watchObservedRunningTime="2026-09-18 06:40:41.450674567 +0000 UTC m=+1121.034060222"
Sep 18 06:40:42.443633 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:42.443596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" event={"ID":"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0","Type":"ContainerStarted","Data":"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"}
Sep 18 06:40:42.444126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:42.443838    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:42.460569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:42.460512    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" podStartSLOduration=3.114997013 podStartE2EDuration="3.460497678s" totalImagesPullingTime="345.500665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:42.459046545 +0000 UTC m=+1122.042432199" watchObservedRunningTime="2026-09-18 06:40:42.460497678 +0000 UTC m=+1122.043883355"
Sep 18 06:40:46.439855 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:46.439826    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:40:48.455337 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:48.455308    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:40:52.445333 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:52.445303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:40:54.636807 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.636772    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:40:54.640603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.640584    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.643615 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.643587    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-708b6f-dockercfg-84fpd\""
Sep 18 06:40:54.643711 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.643632    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-708b6f\""
Sep 18 06:40:54.649161 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.648960    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:40:54.735710 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.735680    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.735889 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.735817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886pc\" (UniqueName: \"kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.837210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.837132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886pc\" (UniqueName: \"kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.837368 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.837256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.837895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.837877    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.845049 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.845018    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-886pc\" (UniqueName: \"kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc\") pod \"payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:54.953525 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:54.953491    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:55.224639 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:55.224614    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:40:55.226421 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:40:55.226395    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8377056_7c62_491f_b5f1_257d6a3af96f.slice/crio-9975f81d5e5dbba5fee56c85ddf7062cd69c3f3afd6b5b4f6ce303055fb76b57 WatchSource:0}: Error finding container 9975f81d5e5dbba5fee56c85ddf7062cd69c3f3afd6b5b4f6ce303055fb76b57: Status 404 returned error can't find the container with id 9975f81d5e5dbba5fee56c85ddf7062cd69c3f3afd6b5b4f6ce303055fb76b57
Sep 18 06:40:55.517267 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:55.517183    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" event={"ID":"e8377056-7c62-491f-b5f1-257d6a3af96f","Type":"ContainerStarted","Data":"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"}
Sep 18 06:40:55.517267 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:55.517237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" event={"ID":"e8377056-7c62-491f-b5f1-257d6a3af96f","Type":"ContainerStarted","Data":"9975f81d5e5dbba5fee56c85ddf7062cd69c3f3afd6b5b4f6ce303055fb76b57"}
Sep 18 06:40:55.517446 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:55.517280    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:40:55.534365 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:40:55.534314    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" podStartSLOduration=1.534299944 podStartE2EDuration="1.534299944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:40:55.533582927 +0000 UTC m=+1135.116968581" watchObservedRunningTime="2026-09-18 06:40:55.534299944 +0000 UTC m=+1135.117685597"
Sep 18 06:41:01.480940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.480902    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:41:01.481583 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.481177    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" podUID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" containerName="maas-api" containerID="cri-o://1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85" gracePeriod=30
Sep 18 06:41:01.737745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.737682    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:41:01.904663 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.904632    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca\") pod \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") "
Sep 18 06:41:01.904833 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.904705    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs\") pod \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") "
Sep 18 06:41:01.904833 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.904760    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxch9\" (UniqueName: \"kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9\") pod \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") "
Sep 18 06:41:01.904833 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.904785    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") pod \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\" (UID: \"af83dfd5-f6ce-4faa-a287-4efc9afc167e\") "
Sep 18 06:41:01.905010 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.904981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af83dfd5-f6ce-4faa-a287-4efc9afc167e" (UID: "af83dfd5-f6ce-4faa-a287-4efc9afc167e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:41:01.906980 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.906948    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af83dfd5-f6ce-4faa-a287-4efc9afc167e" (UID: "af83dfd5-f6ce-4faa-a287-4efc9afc167e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:41:01.907104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.907000    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af83dfd5-f6ce-4faa-a287-4efc9afc167e" (UID: "af83dfd5-f6ce-4faa-a287-4efc9afc167e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:41:01.907104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:01.907029    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9" (OuterVolumeSpecName: "kube-api-access-gxch9") pod "af83dfd5-f6ce-4faa-a287-4efc9afc167e" (UID: "af83dfd5-f6ce-4faa-a287-4efc9afc167e"). InnerVolumeSpecName "kube-api-access-gxch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:41:02.005519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.005416    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af83dfd5-f6ce-4faa-a287-4efc9afc167e-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:02.005519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.005453    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:02.005519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.005465    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxch9\" (UniqueName: \"kubernetes.io/projected/af83dfd5-f6ce-4faa-a287-4efc9afc167e-kube-api-access-gxch9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:02.005519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.005478    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af83dfd5-f6ce-4faa-a287-4efc9afc167e-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:02.557465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.557429    2567 generic.go:356] "Generic (PLEG): container finished" podID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" containerID="1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85" exitCode=0
Sep 18 06:41:02.557879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.557474    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" event={"ID":"af83dfd5-f6ce-4faa-a287-4efc9afc167e","Type":"ContainerDied","Data":"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"}
Sep 18 06:41:02.557879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.557479    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"
Sep 18 06:41:02.557879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.557519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz" event={"ID":"af83dfd5-f6ce-4faa-a287-4efc9afc167e","Type":"ContainerDied","Data":"92f087bebc819414d11a0e8815398629c479e81ff6ea8101b3b64039db67479e"}
Sep 18 06:41:02.557879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.557542    2567 scope.go:117] "RemoveContainer" containerID="1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"
Sep 18 06:41:02.567637 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.567617    2567 scope.go:117] "RemoveContainer" containerID="1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"
Sep 18 06:41:02.567913 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:41:02.567894    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85\": container with ID starting with 1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85 not found: ID does not exist" containerID="1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"
Sep 18 06:41:02.567966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.567923    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85"} err="failed to get container status \"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85\": rpc error: code = NotFound desc = could not find container \"1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85\": container with ID starting with 1b535833802ca6f98e35c2bc431ccdfef987e51833b35e2458e27492d71aaf85 not found: ID does not exist"
Sep 18 06:41:02.578802 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.578777    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:41:02.583923 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.583895    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0da33e9c-5bbdfc646d-r6nhz"]
Sep 18 06:41:02.909828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:02.909741    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" path="/var/lib/kubelet/pods/af83dfd5-f6ce-4faa-a287-4efc9afc167e/volumes"
Sep 18 06:41:06.525095 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.525064    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:41:06.956728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.956694    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:06.957348 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.957330    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" containerName="maas-api"
Sep 18 06:41:06.957414 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.957352    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" containerName="maas-api"
Sep 18 06:41:06.957464 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.957452    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="af83dfd5-f6ce-4faa-a287-4efc9afc167e" containerName="maas-api"
Sep 18 06:41:06.962323 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.962302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:06.964691 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.964674    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-994a4fd1-serving-cert\""
Sep 18 06:41:06.969329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:06.969304    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:07.054732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.054700    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxv5\" (UniqueName: \"kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.054901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.054775    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.054901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.054818    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.054901 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.054869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.155416 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.155373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.155584 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.155443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxv5\" (UniqueName: \"kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.155584 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.155505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.155584 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:41:07.155522    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-serving-cert: secret "maas-api-e2e-migrate-994a4fd1-serving-cert" not found
Sep 18 06:41:07.155584 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.155559    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.155584 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:41:07.155578    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls podName:2acdde16-27f8-46a4-b272-db2a5702f5bc nodeName:}" failed. No retries permitted until 2026-09-18 06:41:07.655563523 +0000 UTC m=+1147.238949159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls") pod "maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" (UID: "2acdde16-27f8-46a4-b272-db2a5702f5bc") : secret "maas-api-e2e-migrate-994a4fd1-serving-cert" not found
Sep 18 06:41:07.156211 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.156190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.157976 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.157952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.163679 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.163656    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxv5\" (UniqueName: \"kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.659042 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.659006    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.661653 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.661621    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") pod \"maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:07.749094 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.749063    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:07.752907 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.752892    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.755433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.755417    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-994a4fd1-dockercfg-mmr6r\""
Sep 18 06:41:07.755495 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.755420    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-994a4fd1\""
Sep 18 06:41:07.759770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.759741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.759904 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.759844    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmpf\" (UniqueName: \"kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.762857 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.762834    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:07.860425 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.860390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.860602 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.860436    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmpf\" (UniqueName: \"kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.861047 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.861028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.868370 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.868335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmpf\" (UniqueName: \"kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf\") pod \"payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") " pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:07.875011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:07.874987    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:08.064940 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.064907    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:08.144409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.144378    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:08.146515 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:41:08.146485    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acdde16_27f8_46a4_b272_db2a5702f5bc.slice/crio-5c252431a956da1f315718bbeac6b85c312d4a1abe078d5884cfce4e505ee999 WatchSource:0}: Error finding container 5c252431a956da1f315718bbeac6b85c312d4a1abe078d5884cfce4e505ee999: Status 404 returned error can't find the container with id 5c252431a956da1f315718bbeac6b85c312d4a1abe078d5884cfce4e505ee999
Sep 18 06:41:08.333605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.333580    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:08.336083 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:41:08.336053    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3449d6e_81b8_47b3_8a1c_757c83cedf75.slice/crio-8f2a864190b0f661915aa0deb0d9ecfb45af57d3bba3c77067a0a1ab480af2bb WatchSource:0}: Error finding container 8f2a864190b0f661915aa0deb0d9ecfb45af57d3bba3c77067a0a1ab480af2bb: Status 404 returned error can't find the container with id 8f2a864190b0f661915aa0deb0d9ecfb45af57d3bba3c77067a0a1ab480af2bb
Sep 18 06:41:08.591390 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.591318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" event={"ID":"b3449d6e-81b8-47b3-8a1c-757c83cedf75","Type":"ContainerStarted","Data":"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"}
Sep 18 06:41:08.591501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.591418    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" event={"ID":"b3449d6e-81b8-47b3-8a1c-757c83cedf75","Type":"ContainerStarted","Data":"8f2a864190b0f661915aa0deb0d9ecfb45af57d3bba3c77067a0a1ab480af2bb"}
Sep 18 06:41:08.591501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.591448    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:08.593072 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.593045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" event={"ID":"2acdde16-27f8-46a4-b272-db2a5702f5bc","Type":"ContainerStarted","Data":"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"}
Sep 18 06:41:08.593183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.593094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" event={"ID":"2acdde16-27f8-46a4-b272-db2a5702f5bc","Type":"ContainerStarted","Data":"5c252431a956da1f315718bbeac6b85c312d4a1abe078d5884cfce4e505ee999"}
Sep 18 06:41:08.593183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.593126    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:08.609386 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.609328    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" podStartSLOduration=1.609310115 podStartE2EDuration="1.609310115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:08.605724227 +0000 UTC m=+1148.189109881" watchObservedRunningTime="2026-09-18 06:41:08.609310115 +0000 UTC m=+1148.192695771"
Sep 18 06:41:08.622601 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:08.622554    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" podStartSLOduration=2.25699054 podStartE2EDuration="2.622538576s" totalImagesPullingTime="365.548036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:08.620989654 +0000 UTC m=+1148.204375308" watchObservedRunningTime="2026-09-18 06:41:08.622538576 +0000 UTC m=+1148.205924228"
Sep 18 06:41:14.603566 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:14.603537    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:19.599898 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:19.599864    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:47.031917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.031879    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:47.033567 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.033493    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" podUID="2acdde16-27f8-46a4-b272-db2a5702f5bc" containerName="maas-api" containerID="cri-o://256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752" gracePeriod=30
Sep 18 06:41:47.085266 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.085229    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:47.085561 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.085503    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" podUID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" containerName="payload-processing" containerID="cri-o://da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910" gracePeriod=30
Sep 18 06:41:47.337922 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.337892    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:47.391489 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.391463    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:47.437157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzxv5\" (UniqueName: \"kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5\") pod \"2acdde16-27f8-46a4-b272-db2a5702f5bc\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") "
Sep 18 06:41:47.437341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437182    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs\") pod \"2acdde16-27f8-46a4-b272-db2a5702f5bc\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") "
Sep 18 06:41:47.437341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437218    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume\") pod \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") "
Sep 18 06:41:47.437341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437268    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca\") pod \"2acdde16-27f8-46a4-b272-db2a5702f5bc\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") "
Sep 18 06:41:47.437341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") pod \"2acdde16-27f8-46a4-b272-db2a5702f5bc\" (UID: \"2acdde16-27f8-46a4-b272-db2a5702f5bc\") "
Sep 18 06:41:47.437341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437326    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wmpf\" (UniqueName: \"kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf\") pod \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\" (UID: \"b3449d6e-81b8-47b3-8a1c-757c83cedf75\") "
Sep 18 06:41:47.437674 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437641    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2acdde16-27f8-46a4-b272-db2a5702f5bc" (UID: "2acdde16-27f8-46a4-b272-db2a5702f5bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:41:47.437674 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.437659    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3449d6e-81b8-47b3-8a1c-757c83cedf75" (UID: "b3449d6e-81b8-47b3-8a1c-757c83cedf75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:41:47.439625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.439596    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2acdde16-27f8-46a4-b272-db2a5702f5bc" (UID: "2acdde16-27f8-46a4-b272-db2a5702f5bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:41:47.439783 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.439767    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2acdde16-27f8-46a4-b272-db2a5702f5bc" (UID: "2acdde16-27f8-46a4-b272-db2a5702f5bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:41:47.439920 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.439904    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf" (OuterVolumeSpecName: "kube-api-access-5wmpf") pod "b3449d6e-81b8-47b3-8a1c-757c83cedf75" (UID: "b3449d6e-81b8-47b3-8a1c-757c83cedf75"). InnerVolumeSpecName "kube-api-access-5wmpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:41:47.439966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.439923    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5" (OuterVolumeSpecName: "kube-api-access-pzxv5") pod "2acdde16-27f8-46a4-b272-db2a5702f5bc" (UID: "2acdde16-27f8-46a4-b272-db2a5702f5bc"). InnerVolumeSpecName "kube-api-access-pzxv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:41:47.537964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537921    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acdde16-27f8-46a4-b272-db2a5702f5bc-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.537964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537951    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.537964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537960    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmpf\" (UniqueName: \"kubernetes.io/projected/b3449d6e-81b8-47b3-8a1c-757c83cedf75-kube-api-access-5wmpf\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.537964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537969    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxv5\" (UniqueName: \"kubernetes.io/projected/2acdde16-27f8-46a4-b272-db2a5702f5bc-kube-api-access-pzxv5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.537964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537980    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2acdde16-27f8-46a4-b272-db2a5702f5bc-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.538396 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.537988    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3449d6e-81b8-47b3-8a1c-757c83cedf75-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:41:47.788743 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.788702    2567 generic.go:356] "Generic (PLEG): container finished" podID="2acdde16-27f8-46a4-b272-db2a5702f5bc" containerID="256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752" exitCode=0
Sep 18 06:41:47.788925 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.788757    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"
Sep 18 06:41:47.788925 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.788791    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" event={"ID":"2acdde16-27f8-46a4-b272-db2a5702f5bc","Type":"ContainerDied","Data":"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"}
Sep 18 06:41:47.788925 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.788845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl" event={"ID":"2acdde16-27f8-46a4-b272-db2a5702f5bc","Type":"ContainerDied","Data":"5c252431a956da1f315718bbeac6b85c312d4a1abe078d5884cfce4e505ee999"}
Sep 18 06:41:47.788925 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.788868    2567 scope.go:117] "RemoveContainer" containerID="256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"
Sep 18 06:41:47.790301 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.790271    2567 generic.go:356] "Generic (PLEG): container finished" podID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" containerID="da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910" exitCode=0
Sep 18 06:41:47.790408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.790309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" event={"ID":"b3449d6e-81b8-47b3-8a1c-757c83cedf75","Type":"ContainerDied","Data":"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"}
Sep 18 06:41:47.790408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.790351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf" event={"ID":"b3449d6e-81b8-47b3-8a1c-757c83cedf75","Type":"ContainerDied","Data":"8f2a864190b0f661915aa0deb0d9ecfb45af57d3bba3c77067a0a1ab480af2bb"}
Sep 18 06:41:47.790408 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.790320    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"
Sep 18 06:41:47.799091 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.799072    2567 scope.go:117] "RemoveContainer" containerID="256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"
Sep 18 06:41:47.799506 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:41:47.799474    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752\": container with ID starting with 256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752 not found: ID does not exist" containerID="256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"
Sep 18 06:41:47.799607 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.799511    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752"} err="failed to get container status \"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752\": rpc error: code = NotFound desc = could not find container \"256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752\": container with ID starting with 256afce28df9b4119fcf1f4c1feb26c2ba1b1ae61578169bf67659cb270aa752 not found: ID does not exist"
Sep 18 06:41:47.799607 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.799534    2567 scope.go:117] "RemoveContainer" containerID="da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"
Sep 18 06:41:47.807425 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.807404    2567 scope.go:117] "RemoveContainer" containerID="da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"
Sep 18 06:41:47.807674 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:41:47.807656    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910\": container with ID starting with da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910 not found: ID does not exist" containerID="da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"
Sep 18 06:41:47.807728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.807681    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910"} err="failed to get container status \"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910\": rpc error: code = NotFound desc = could not find container \"da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910\": container with ID starting with da9f1a6be5319126c80bbc2b5fef462b45c9f3e2b5d60389ff4882f1494ed910 not found: ID does not exist"
Sep 18 06:41:47.811065 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.811045    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:47.813023 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.813002    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-994a4fd1-6878f5c47c-5nfwl"]
Sep 18 06:41:47.821741 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.821708    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:47.824791 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:47.824770    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-994a4fd1-795478bffd-whvsf"]
Sep 18 06:41:48.909297 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:48.909265    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2acdde16-27f8-46a4-b272-db2a5702f5bc" path="/var/lib/kubelet/pods/2acdde16-27f8-46a4-b272-db2a5702f5bc/volumes"
Sep 18 06:41:48.909713 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:48.909634    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" path="/var/lib/kubelet/pods/b3449d6e-81b8-47b3-8a1c-757c83cedf75/volumes"
Sep 18 06:41:52.957941 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.957910    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:41:52.958339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958325    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" containerName="payload-processing"
Sep 18 06:41:52.958387 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958339    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" containerName="payload-processing"
Sep 18 06:41:52.958387 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958359    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acdde16-27f8-46a4-b272-db2a5702f5bc" containerName="maas-api"
Sep 18 06:41:52.958387 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958365    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acdde16-27f8-46a4-b272-db2a5702f5bc" containerName="maas-api"
Sep 18 06:41:52.958501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958434    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3449d6e-81b8-47b3-8a1c-757c83cedf75" containerName="payload-processing"
Sep 18 06:41:52.958501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.958441    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2acdde16-27f8-46a4-b272-db2a5702f5bc" containerName="maas-api"
Sep 18 06:41:52.963148 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.963118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:52.965494 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.965472    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5e4c6960-dockercfg-hb4w8\""
Sep 18 06:41:52.965602 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.965570    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5e4c6960\""
Sep 18 06:41:52.973264 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.973238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:41:52.977356 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.977335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:52.977463 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:52.977387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6scl\" (UniqueName: \"kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.078283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.078247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.078283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.078297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6scl\" (UniqueName: \"kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.078956 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.078931    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.086019 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.085992    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6scl\" (UniqueName: \"kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl\") pod \"payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") " pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.275201 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.275101    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.546697 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.546671    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:41:53.548755 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:41:53.548725    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57134233_6878_4942_bf35_e7d6df13ed52.slice/crio-01699ced8cd8137b8e029823dea32ad37fc32e3e9ebf0554a7b4bc498df9b325 WatchSource:0}: Error finding container 01699ced8cd8137b8e029823dea32ad37fc32e3e9ebf0554a7b4bc498df9b325: Status 404 returned error can't find the container with id 01699ced8cd8137b8e029823dea32ad37fc32e3e9ebf0554a7b4bc498df9b325
Sep 18 06:41:53.827029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.826939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" event={"ID":"57134233-6878-4942-bf35-e7d6df13ed52","Type":"ContainerStarted","Data":"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"}
Sep 18 06:41:53.827029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.826999    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" event={"ID":"57134233-6878-4942-bf35-e7d6df13ed52","Type":"ContainerStarted","Data":"01699ced8cd8137b8e029823dea32ad37fc32e3e9ebf0554a7b4bc498df9b325"}
Sep 18 06:41:53.827029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.827028    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:41:53.845839 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:41:53.845776    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" podStartSLOduration=1.845756503 podStartE2EDuration="1.845756503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:41:53.843964184 +0000 UTC m=+1193.427349837" watchObservedRunningTime="2026-09-18 06:41:53.845756503 +0000 UTC m=+1193.429142157"
Sep 18 06:42:00.890595 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:00.890565    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:42:00.892290 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:00.892267    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:42:04.835084 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:04.835047    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:42:31.979966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:31.979931    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:42:31.982505 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:31.980212    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" podUID="57134233-6878-4942-bf35-e7d6df13ed52" containerName="payload-processing" containerID="cri-o://4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40" gracePeriod=30
Sep 18 06:42:32.234909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.234843    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:42:32.330787 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.330751    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6scl\" (UniqueName: \"kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl\") pod \"57134233-6878-4942-bf35-e7d6df13ed52\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") "
Sep 18 06:42:32.330969 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.330843    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume\") pod \"57134233-6878-4942-bf35-e7d6df13ed52\" (UID: \"57134233-6878-4942-bf35-e7d6df13ed52\") "
Sep 18 06:42:32.331265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.331239    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57134233-6878-4942-bf35-e7d6df13ed52" (UID: "57134233-6878-4942-bf35-e7d6df13ed52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:42:32.332850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.332830    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl" (OuterVolumeSpecName: "kube-api-access-l6scl") pod "57134233-6878-4942-bf35-e7d6df13ed52" (UID: "57134233-6878-4942-bf35-e7d6df13ed52"). InnerVolumeSpecName "kube-api-access-l6scl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:42:32.431647 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.431611    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6scl\" (UniqueName: \"kubernetes.io/projected/57134233-6878-4942-bf35-e7d6df13ed52-kube-api-access-l6scl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:42:32.431647 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:32.431642    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57134233-6878-4942-bf35-e7d6df13ed52-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:42:33.030185 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.030117    2567 generic.go:356] "Generic (PLEG): container finished" podID="57134233-6878-4942-bf35-e7d6df13ed52" containerID="4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40" exitCode=0
Sep 18 06:42:33.030723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.030202    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"
Sep 18 06:42:33.030723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.030201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" event={"ID":"57134233-6878-4942-bf35-e7d6df13ed52","Type":"ContainerDied","Data":"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"}
Sep 18 06:42:33.030723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.030462    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz" event={"ID":"57134233-6878-4942-bf35-e7d6df13ed52","Type":"ContainerDied","Data":"01699ced8cd8137b8e029823dea32ad37fc32e3e9ebf0554a7b4bc498df9b325"}
Sep 18 06:42:33.030723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.030555    2567 scope.go:117] "RemoveContainer" containerID="4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"
Sep 18 06:42:33.039870 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.039851    2567 scope.go:117] "RemoveContainer" containerID="4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"
Sep 18 06:42:33.040165 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:42:33.040129    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40\": container with ID starting with 4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40 not found: ID does not exist" containerID="4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"
Sep 18 06:42:33.040238 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.040174    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40"} err="failed to get container status \"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40\": rpc error: code = NotFound desc = could not find container \"4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40\": container with ID starting with 4e6b11a9039a5f8db01915bd02f40529b366fa32747f437d02ef00cb7f539a40 not found: ID does not exist"
Sep 18 06:42:33.045751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.045731    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:42:33.049502 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:33.049480    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5e4c6960-7dd6957f4c-2sjlz"]
Sep 18 06:42:34.911426 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:34.911392    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57134233-6878-4942-bf35-e7d6df13ed52" path="/var/lib/kubelet/pods/57134233-6878-4942-bf35-e7d6df13ed52/volumes"
Sep 18 06:42:40.447221 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.447185    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:42:40.448519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.448490    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57134233-6878-4942-bf35-e7d6df13ed52" containerName="payload-processing"
Sep 18 06:42:40.448519 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.448519    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="57134233-6878-4942-bf35-e7d6df13ed52" containerName="payload-processing"
Sep 18 06:42:40.448944 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.448919    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="57134233-6878-4942-bf35-e7d6df13ed52" containerName="payload-processing"
Sep 18 06:42:40.456605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.456579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.459711 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.459688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-246af4f6-serving-cert\""
Sep 18 06:42:40.462469 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.462437    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:42:40.606212 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.606178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.606212 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.606210    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kq6\" (UniqueName: \"kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.606470 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.606236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.606470 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.606386    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.707264 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.707183    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.707264 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.707216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kq6\" (UniqueName: \"kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.707264 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.707237    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.707481 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.707292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.707987 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.707969    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.709548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.709522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.709669 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.709569    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.714436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.714413    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kq6\" (UniqueName: \"kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6\") pod \"maas-api-e2e-derive-246af4f6-6496579d7b-8fphj\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:40.769987 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:40.769952    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:41.041550 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:41.041524    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:42:41.043530 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:42:41.043494    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f43b3d_82a6_4a9e_9a70_5ae65615fd62.slice/crio-5d20ddb8636c81c50ff58f7fb9478e4df372be57843af57be6c6bc1d57bf33b3 WatchSource:0}: Error finding container 5d20ddb8636c81c50ff58f7fb9478e4df372be57843af57be6c6bc1d57bf33b3: Status 404 returned error can't find the container with id 5d20ddb8636c81c50ff58f7fb9478e4df372be57843af57be6c6bc1d57bf33b3
Sep 18 06:42:41.044876 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:41.044858    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:42:41.076111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:41.076078    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" event={"ID":"87f43b3d-82a6-4a9e-9a70-5ae65615fd62","Type":"ContainerStarted","Data":"5d20ddb8636c81c50ff58f7fb9478e4df372be57843af57be6c6bc1d57bf33b3"}
Sep 18 06:42:42.082426 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:42.082385    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" event={"ID":"87f43b3d-82a6-4a9e-9a70-5ae65615fd62","Type":"ContainerStarted","Data":"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"}
Sep 18 06:42:42.082802 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:42.082508    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:42:42.099335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:42.099286    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" podStartSLOduration=1.7532202780000001 podStartE2EDuration="2.099267771s" totalImagesPullingTime="346.047493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:42:42.097781789 +0000 UTC m=+1241.681167442" watchObservedRunningTime="2026-09-18 06:42:42.099267771 +0000 UTC m=+1241.682653425"
Sep 18 06:42:48.092718 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:42:48.092690    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:43:05.079854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.079816    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:43:05.084899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.084874    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.087585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.087560    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c4f4b\"/\"kube-root-ca.crt\""
Sep 18 06:43:05.088697 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.088644    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c4f4b\"/\"default-dockercfg-sm7mt\""
Sep 18 06:43:05.088820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.088800    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7c4f4b\"/\"openshift-service-ca.crt\""
Sep 18 06:43:05.088891 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.088797    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7c4f4b\"/\"test-model-7c4f4b-kserve-self-signed-certs\""
Sep 18 06:43:05.095589 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.095563    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:43:05.117707 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.117899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117714    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.117899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjnl4\" (UniqueName: \"kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.117899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.117899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.118074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.117902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.218588 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjnl4\" (UniqueName: \"kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.218588 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.218853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.218853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.218853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.219046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.219046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.218945    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.219115 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.219078    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.219194 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.219175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.221192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.221132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.221400 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.221372    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.226334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.226309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjnl4\" (UniqueName: \"kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4\") pod \"test-model-7c4f4b-kserve-58ff78f9b9-8gwzk\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") " pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.396208 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.396117    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:05.669932 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:05.669904    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:43:05.671819 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:43:05.671781    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9de02f9_24cf_4045_b81b_aec16e797401.slice/crio-4be07f8eadfe1fa62681a2619e5717b5925b051843f54470a97218c0b02c2764 WatchSource:0}: Error finding container 4be07f8eadfe1fa62681a2619e5717b5925b051843f54470a97218c0b02c2764: Status 404 returned error can't find the container with id 4be07f8eadfe1fa62681a2619e5717b5925b051843f54470a97218c0b02c2764
Sep 18 06:43:06.207574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:06.207534    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" event={"ID":"e9de02f9-24cf-4045-b81b-aec16e797401","Type":"ContainerStarted","Data":"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"}
Sep 18 06:43:06.208007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:06.207585    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" event={"ID":"e9de02f9-24cf-4045-b81b-aec16e797401","Type":"ContainerStarted","Data":"4be07f8eadfe1fa62681a2619e5717b5925b051843f54470a97218c0b02c2764"}
Sep 18 06:43:06.224468 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:06.224416    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" podStartSLOduration=1.224400138 podStartE2EDuration="1.224400138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:43:06.222122187 +0000 UTC m=+1265.805507842" watchObservedRunningTime="2026-09-18 06:43:06.224400138 +0000 UTC m=+1265.807785791"
Sep 18 06:43:15.396472 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:15.396423    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:15.396917 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:15.396489    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:15.409011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:15.408990    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:16.267909 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:16.267877    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:43:18.609205 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.609169    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:43:18.609669 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.609386    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" podUID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" containerName="maas-api" containerID="cri-o://11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9" gracePeriod=30
Sep 18 06:43:18.871537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.871465    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:43:18.958687 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.958658    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls\") pod \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") "
Sep 18 06:43:18.958880 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.958713    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs\") pod \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") "
Sep 18 06:43:18.958880 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.958746    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca\") pod \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") "
Sep 18 06:43:18.959001 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.958929    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89kq6\" (UniqueName: \"kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6\") pod \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\" (UID: \"87f43b3d-82a6-4a9e-9a70-5ae65615fd62\") "
Sep 18 06:43:18.959198 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.959168    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87f43b3d-82a6-4a9e-9a70-5ae65615fd62" (UID: "87f43b3d-82a6-4a9e-9a70-5ae65615fd62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:43:18.959348 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.959331    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:18.960910 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.960882    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87f43b3d-82a6-4a9e-9a70-5ae65615fd62" (UID: "87f43b3d-82a6-4a9e-9a70-5ae65615fd62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:43:18.960986 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.960903    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87f43b3d-82a6-4a9e-9a70-5ae65615fd62" (UID: "87f43b3d-82a6-4a9e-9a70-5ae65615fd62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:43:18.960986 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:18.960921    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6" (OuterVolumeSpecName: "kube-api-access-89kq6") pod "87f43b3d-82a6-4a9e-9a70-5ae65615fd62" (UID: "87f43b3d-82a6-4a9e-9a70-5ae65615fd62"). InnerVolumeSpecName "kube-api-access-89kq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:43:19.060890 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.060850    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:19.060890 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.060891    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:19.061127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.060919    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89kq6\" (UniqueName: \"kubernetes.io/projected/87f43b3d-82a6-4a9e-9a70-5ae65615fd62-kube-api-access-89kq6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:19.273876 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.273839    2567 generic.go:356] "Generic (PLEG): container finished" podID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" containerID="11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9" exitCode=0
Sep 18 06:43:19.274054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.273885    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"
Sep 18 06:43:19.274054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.273931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" event={"ID":"87f43b3d-82a6-4a9e-9a70-5ae65615fd62","Type":"ContainerDied","Data":"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"}
Sep 18 06:43:19.274054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.273985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj" event={"ID":"87f43b3d-82a6-4a9e-9a70-5ae65615fd62","Type":"ContainerDied","Data":"5d20ddb8636c81c50ff58f7fb9478e4df372be57843af57be6c6bc1d57bf33b3"}
Sep 18 06:43:19.274054 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.274009    2567 scope.go:117] "RemoveContainer" containerID="11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"
Sep 18 06:43:19.284108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.284093    2567 scope.go:117] "RemoveContainer" containerID="11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"
Sep 18 06:43:19.284401 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:43:19.284383    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9\": container with ID starting with 11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9 not found: ID does not exist" containerID="11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"
Sep 18 06:43:19.284453 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.284411    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9"} err="failed to get container status \"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9\": rpc error: code = NotFound desc = could not find container \"11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9\": container with ID starting with 11e3b180fb85b9688dd5a43b122c513c55ad610aeed0a1d5e4a4fd4e1e9f3ef9 not found: ID does not exist"
Sep 18 06:43:19.295810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.295778    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:43:19.299433 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:19.299412    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-246af4f6-6496579d7b-8fphj"]
Sep 18 06:43:20.909545 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:20.909512    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" path="/var/lib/kubelet/pods/87f43b3d-82a6-4a9e-9a70-5ae65615fd62/volumes"
Sep 18 06:43:34.320699 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.320661    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:43:34.321113 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.320970    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" podUID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" containerName="maas-api" containerID="cri-o://eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18" gracePeriod=30
Sep 18 06:43:34.369311 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.369276    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:43:34.369535 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.369503    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" podUID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" containerName="payload-pre-processing" containerID="cri-o://11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38" gracePeriod=30
Sep 18 06:43:34.615617 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.615588    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:43:34.650046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.650023    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:43:34.705823 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.705795    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs\") pod \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") "
Sep 18 06:43:34.706017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.705868    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume\") pod \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") "
Sep 18 06:43:34.706017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.705961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") pod \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") "
Sep 18 06:43:34.706192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.706000    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9rn7\" (UniqueName: \"kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7\") pod \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\" (UID: \"68d336b1-9ce2-4293-adf7-9f8e2ffead8c\") "
Sep 18 06:43:34.706563 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.706414    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca\") pod \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") "
Sep 18 06:43:34.706563 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.706503    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt4sk\" (UniqueName: \"kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk\") pod \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\" (UID: \"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0\") "
Sep 18 06:43:34.706712 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.706552    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68d336b1-9ce2-4293-adf7-9f8e2ffead8c" (UID: "68d336b1-9ce2-4293-adf7-9f8e2ffead8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:43:34.707019 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.706994    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" (UID: "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:43:34.707673 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.707649    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:34.707770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.707678    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:34.709009 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.708972    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" (UID: "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:43:34.709097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.709062    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk" (OuterVolumeSpecName: "kube-api-access-xt4sk") pod "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" (UID: "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0"). InnerVolumeSpecName "kube-api-access-xt4sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:43:34.709169 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.709106    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7" (OuterVolumeSpecName: "kube-api-access-q9rn7") pod "68d336b1-9ce2-4293-adf7-9f8e2ffead8c" (UID: "68d336b1-9ce2-4293-adf7-9f8e2ffead8c"). InnerVolumeSpecName "kube-api-access-q9rn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:43:34.712193 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.709326    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" (UID: "fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:43:34.809069 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.809033    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:34.809069 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.809064    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rn7\" (UniqueName: \"kubernetes.io/projected/68d336b1-9ce2-4293-adf7-9f8e2ffead8c-kube-api-access-q9rn7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:34.809069 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.809076    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt4sk\" (UniqueName: \"kubernetes.io/projected/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-kube-api-access-xt4sk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:34.809308 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:34.809086    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:43:35.362758 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.362725    2567 generic.go:356] "Generic (PLEG): container finished" podID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" containerID="eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18" exitCode=0
Sep 18 06:43:35.363200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.362782    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"
Sep 18 06:43:35.363200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.362815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" event={"ID":"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0","Type":"ContainerDied","Data":"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"}
Sep 18 06:43:35.363200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.362864    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb" event={"ID":"fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0","Type":"ContainerDied","Data":"856786d2b2c0f4bf392ae124af0d75487eede0feb90bf1ce4ecfab0b8ba482f4"}
Sep 18 06:43:35.363200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.362890    2567 scope.go:117] "RemoveContainer" containerID="eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"
Sep 18 06:43:35.364431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.364289    2567 generic.go:356] "Generic (PLEG): container finished" podID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" containerID="11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38" exitCode=0
Sep 18 06:43:35.364431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.364393    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" event={"ID":"68d336b1-9ce2-4293-adf7-9f8e2ffead8c","Type":"ContainerDied","Data":"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"}
Sep 18 06:43:35.364431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.364426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j" event={"ID":"68d336b1-9ce2-4293-adf7-9f8e2ffead8c","Type":"ContainerDied","Data":"30f324968f1156e6e8e2e8ed266d093e70540fae9645ef43552e29c4d84d8410"}
Sep 18 06:43:35.364617 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.364512    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"
Sep 18 06:43:35.372236 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.372219    2567 scope.go:117] "RemoveContainer" containerID="eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"
Sep 18 06:43:35.372565 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:43:35.372537    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18\": container with ID starting with eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18 not found: ID does not exist" containerID="eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"
Sep 18 06:43:35.372677 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.372569    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18"} err="failed to get container status \"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18\": rpc error: code = NotFound desc = could not find container \"eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18\": container with ID starting with eb644b93b94aa346ff67862af9b84ffc926fa2532ae7a4e3f54c0b66b8c09e18 not found: ID does not exist"
Sep 18 06:43:35.372677 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.372590    2567 scope.go:117] "RemoveContainer" containerID="11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"
Sep 18 06:43:35.380994 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.380978    2567 scope.go:117] "RemoveContainer" containerID="11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"
Sep 18 06:43:35.381248 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:43:35.381231    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38\": container with ID starting with 11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38 not found: ID does not exist" containerID="11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"
Sep 18 06:43:35.381309 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.381253    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38"} err="failed to get container status \"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38\": rpc error: code = NotFound desc = could not find container \"11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38\": container with ID starting with 11406dffa7bc7985f84e8bb33752457f57a91f1794ad9bf1679d987ee26f4e38 not found: ID does not exist"
Sep 18 06:43:35.386124 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.386102    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:43:35.389470 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.389450    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-43fdfe-86b8cdc479-b48pb"]
Sep 18 06:43:35.401677 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.401653    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:43:35.405331 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:35.405305    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-43fdfe-7f95cfd466-fdq2j"]
Sep 18 06:43:36.909641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:36.909605    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" path="/var/lib/kubelet/pods/68d336b1-9ce2-4293-adf7-9f8e2ffead8c/volumes"
Sep 18 06:43:36.910111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:43:36.910004    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" path="/var/lib/kubelet/pods/fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0/volumes"
Sep 18 06:44:16.190656 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.190625    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:44:16.191229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191121    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" containerName="payload-pre-processing"
Sep 18 06:44:16.191229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191198    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" containerName="payload-pre-processing"
Sep 18 06:44:16.191229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191219    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" containerName="maas-api"
Sep 18 06:44:16.191229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191228    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" containerName="maas-api"
Sep 18 06:44:16.191409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191272    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" containerName="maas-api"
Sep 18 06:44:16.191409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191283    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" containerName="maas-api"
Sep 18 06:44:16.191409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191375    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f43b3d-82a6-4a9e-9a70-5ae65615fd62" containerName="maas-api"
Sep 18 06:44:16.191409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191382    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3cdb76-8b5b-4e0d-a05d-7084b58e17c0" containerName="maas-api"
Sep 18 06:44:16.191409 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.191388    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d336b1-9ce2-4293-adf7-9f8e2ffead8c" containerName="payload-pre-processing"
Sep 18 06:44:16.195914 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.195896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.199756 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.199732    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-946609\"/\"kube-root-ca.crt\""
Sep 18 06:44:16.199881 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.199787    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-946609\"/\"openshift-service-ca.crt\""
Sep 18 06:44:16.199881 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.199730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-946609\"/\"premium-w0-946609-kserve-self-signed-certs\""
Sep 18 06:44:16.200006 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.199975    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-946609\"/\"default-dockercfg-sbzpb\""
Sep 18 06:44:16.202664 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.202642    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:44:16.289118 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.289300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25vgt\" (UniqueName: \"kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.289300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.289401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.289401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.289401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.289390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390298 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390298 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25vgt\" (UniqueName: \"kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390454    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.390847 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.390783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.392627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.392606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.392723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.392708    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.398617 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.398598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25vgt\" (UniqueName: \"kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt\") pod \"premium-w0-946609-kserve-5478549b46-blnck\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") " pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.507796 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.507757    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:16.779250 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:16.779224    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:44:16.780942 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:44:16.780915    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod972da67d_dbc6_4268_a99f_1aa010bf796f.slice/crio-bf093d5eebfdac0dffda669949a435e1051e3510358b703e95c8584df3a86d0a WatchSource:0}: Error finding container bf093d5eebfdac0dffda669949a435e1051e3510358b703e95c8584df3a86d0a: Status 404 returned error can't find the container with id bf093d5eebfdac0dffda669949a435e1051e3510358b703e95c8584df3a86d0a
Sep 18 06:44:17.577742 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:17.577705    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" event={"ID":"972da67d-dbc6-4268-a99f-1aa010bf796f","Type":"ContainerStarted","Data":"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"}
Sep 18 06:44:17.578120 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:17.577758    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" event={"ID":"972da67d-dbc6-4268-a99f-1aa010bf796f","Type":"ContainerStarted","Data":"bf093d5eebfdac0dffda669949a435e1051e3510358b703e95c8584df3a86d0a"}
Sep 18 06:44:17.596960 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:17.596911    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" podStartSLOduration=1.5968955839999999 podStartE2EDuration="1.596895584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:17.596006844 +0000 UTC m=+1337.179392496" watchObservedRunningTime="2026-09-18 06:44:17.596895584 +0000 UTC m=+1337.180281237"
Sep 18 06:44:23.738200 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.737723    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:44:23.742100 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.742076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.744066 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.744040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.744167 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.744119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvj6\" (UniqueName: \"kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.744261 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.744244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.744302 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.744273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.744536 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.744517    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-14ee59-serving-cert\""
Sep 18 06:44:23.751907 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.751794    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:44:23.844818 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.844786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.844818 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.844818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.845030 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.844858    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.845030 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.844894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvj6\" (UniqueName: \"kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.845030 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:44:23.844928    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-serving-cert: secret "maas-api-e2e-explicit-14ee59-serving-cert" not found
Sep 18 06:44:23.845158 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:44:23.845035    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls podName:582212b8-cc3d-4de5-b5c2-ad509ffd0bde nodeName:}" failed. No retries permitted until 2026-09-18 06:44:24.345014926 +0000 UTC m=+1343.928400580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls") pod "maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" (UID: "582212b8-cc3d-4de5-b5c2-ad509ffd0bde") : secret "maas-api-e2e-explicit-14ee59-serving-cert" not found
Sep 18 06:44:23.845621 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.845594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.847374 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.847348    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:23.860376 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:23.860354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvj6\" (UniqueName: \"kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:24.349249 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:24.349207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:24.360323 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:24.360282    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") pod \"maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:24.659383 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:24.659291    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:24.942921 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:24.942890    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:44:24.945206 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:44:24.945173    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582212b8_cc3d_4de5_b5c2_ad509ffd0bde.slice/crio-1ac2c28ad6ab1e142ddf3ed8175eb6b4e3389792eb355d52992b9bd1e81f4597 WatchSource:0}: Error finding container 1ac2c28ad6ab1e142ddf3ed8175eb6b4e3389792eb355d52992b9bd1e81f4597: Status 404 returned error can't find the container with id 1ac2c28ad6ab1e142ddf3ed8175eb6b4e3389792eb355d52992b9bd1e81f4597
Sep 18 06:44:25.617565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:25.617463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" event={"ID":"582212b8-cc3d-4de5-b5c2-ad509ffd0bde","Type":"ContainerStarted","Data":"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"}
Sep 18 06:44:25.617565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:25.617534    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:44:25.617565 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:25.617555    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" event={"ID":"582212b8-cc3d-4de5-b5c2-ad509ffd0bde","Type":"ContainerStarted","Data":"1ac2c28ad6ab1e142ddf3ed8175eb6b4e3389792eb355d52992b9bd1e81f4597"}
Sep 18 06:44:25.635446 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:25.635398    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" podStartSLOduration=2.291745152 podStartE2EDuration="2.635378191s" totalImagesPullingTime="343.633039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:44:25.633238277 +0000 UTC m=+1345.216623924" watchObservedRunningTime="2026-09-18 06:44:25.635378191 +0000 UTC m=+1345.218763843"
Sep 18 06:44:26.508226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:26.508186    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:26.508844 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:26.508248    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:26.520653 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:26.520624    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:26.634152 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:26.634113    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:44:31.628547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:44:31.628514    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:45:03.452412 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.452374    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:03.456205 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.456178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.458983 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.458960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d887938f-serving-cert\""
Sep 18 06:45:03.467842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.467815    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:03.629363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.629325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.629540 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.629439    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.629540 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.629516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.629627 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.629547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtqj\" (UniqueName: \"kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.731202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.731097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.731202 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.731189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.731384 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.731217    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.731384 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.731233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtqj\" (UniqueName: \"kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.731384 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:45:03.731318    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-serving-cert: secret "maas-api-e2e-mt-d887938f-serving-cert" not found
Sep 18 06:45:03.731484 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:45:03.731399    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls podName:98496502-c9bc-49e2-8ce1-b9be847384f8 nodeName:}" failed. No retries permitted until 2026-09-18 06:45:04.23137751 +0000 UTC m=+1383.814763146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls") pod "maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" (UID: "98496502-c9bc-49e2-8ce1-b9be847384f8") : secret "maas-api-e2e-mt-d887938f-serving-cert" not found
Sep 18 06:45:03.731885 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.731867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.733754 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.733725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:03.739800 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:03.739773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtqj\" (UniqueName: \"kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:04.084124 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.084031    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:04.087915 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.087891    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.090976 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.090940    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d887938f\""
Sep 18 06:45:04.091276 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.091251    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d887938f-dockercfg-55pl8\""
Sep 18 06:45:04.096348 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.096323    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:04.236323 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.236293    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:04.236491 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.236336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.236491 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.236429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fbf\" (UniqueName: \"kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.238851 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.238828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") pod \"maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:04.337728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.337633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fbf\" (UniqueName: \"kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.337728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.337707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.338514 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.338493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.346947 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.346920    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fbf\" (UniqueName: \"kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf\") pod \"payload-processing-e2e-mt-d887938f-644585b75c-h5xkv\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") " pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.368902 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.368855    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:04.402572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.402522    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.651480 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.651452    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:04.652850 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:45:04.652824    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98496502_c9bc_49e2_8ce1_b9be847384f8.slice/crio-ddcc5c1a3fbe8ceb3769f45378aa17dd7651bdfee98ba01dfeafce84d3b72d5f WatchSource:0}: Error finding container ddcc5c1a3fbe8ceb3769f45378aa17dd7651bdfee98ba01dfeafce84d3b72d5f: Status 404 returned error can't find the container with id ddcc5c1a3fbe8ceb3769f45378aa17dd7651bdfee98ba01dfeafce84d3b72d5f
Sep 18 06:45:04.709604 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.709572    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:04.711913 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:45:04.711887    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81516825_8bda_419c_a736_266b71b67ccf.slice/crio-e9cfa70e88b83f00398fbea686dfad562759e9ca1091c327506d1b787d0b76ce WatchSource:0}: Error finding container e9cfa70e88b83f00398fbea686dfad562759e9ca1091c327506d1b787d0b76ce: Status 404 returned error can't find the container with id e9cfa70e88b83f00398fbea686dfad562759e9ca1091c327506d1b787d0b76ce
Sep 18 06:45:04.823128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.823085    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" event={"ID":"81516825-8bda-419c-a736-266b71b67ccf","Type":"ContainerStarted","Data":"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"}
Sep 18 06:45:04.823339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.823184    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:04.823339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.823213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" event={"ID":"81516825-8bda-419c-a736-266b71b67ccf","Type":"ContainerStarted","Data":"e9cfa70e88b83f00398fbea686dfad562759e9ca1091c327506d1b787d0b76ce"}
Sep 18 06:45:04.824653 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.824619    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" event={"ID":"98496502-c9bc-49e2-8ce1-b9be847384f8","Type":"ContainerStarted","Data":"ddcc5c1a3fbe8ceb3769f45378aa17dd7651bdfee98ba01dfeafce84d3b72d5f"}
Sep 18 06:45:04.838896 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:04.838838    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" podStartSLOduration=0.838819749 podStartE2EDuration="838.819749ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:04.837092665 +0000 UTC m=+1384.420478330" watchObservedRunningTime="2026-09-18 06:45:04.838819749 +0000 UTC m=+1384.422205402"
Sep 18 06:45:05.831506 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:05.831470    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" event={"ID":"98496502-c9bc-49e2-8ce1-b9be847384f8","Type":"ContainerStarted","Data":"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"}
Sep 18 06:45:05.831924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:05.831607    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:05.848907 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:05.848854    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" podStartSLOduration=2.418808333 podStartE2EDuration="2.848837378s" totalImagesPullingTime="430.029045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:45:05.848249369 +0000 UTC m=+1385.431635022" watchObservedRunningTime="2026-09-18 06:45:05.848837378 +0000 UTC m=+1385.432223030"
Sep 18 06:45:11.842355 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:11.842324    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:15.834007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:15.833975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:47.318342 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.318306    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:47.318902 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.318617    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" podUID="98496502-c9bc-49e2-8ce1-b9be847384f8" containerName="maas-api" containerID="cri-o://5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682" gracePeriod=30
Sep 18 06:45:47.374844 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.374811    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:47.375072 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.375024    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" podUID="81516825-8bda-419c-a736-266b71b67ccf" containerName="payload-processing" containerID="cri-o://59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b" gracePeriod=30
Sep 18 06:45:47.647292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.647265    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:47.677254 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.677222    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:47.763414 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763382    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gtqj\" (UniqueName: \"kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj\") pod \"98496502-c9bc-49e2-8ce1-b9be847384f8\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") "
Sep 18 06:45:47.763595 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763437    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume\") pod \"81516825-8bda-419c-a736-266b71b67ccf\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") "
Sep 18 06:45:47.763595 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763559    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs\") pod \"98496502-c9bc-49e2-8ce1-b9be847384f8\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") "
Sep 18 06:45:47.763595 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763591    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") pod \"98496502-c9bc-49e2-8ce1-b9be847384f8\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") "
Sep 18 06:45:47.763758 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca\") pod \"98496502-c9bc-49e2-8ce1-b9be847384f8\" (UID: \"98496502-c9bc-49e2-8ce1-b9be847384f8\") "
Sep 18 06:45:47.763758 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763679    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9fbf\" (UniqueName: \"kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf\") pod \"81516825-8bda-419c-a736-266b71b67ccf\" (UID: \"81516825-8bda-419c-a736-266b71b67ccf\") "
Sep 18 06:45:47.764027 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.763990    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81516825-8bda-419c-a736-266b71b67ccf" (UID: "81516825-8bda-419c-a736-266b71b67ccf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:45:47.764183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.764041    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98496502-c9bc-49e2-8ce1-b9be847384f8" (UID: "98496502-c9bc-49e2-8ce1-b9be847384f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:45:47.765772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.765747    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98496502-c9bc-49e2-8ce1-b9be847384f8" (UID: "98496502-c9bc-49e2-8ce1-b9be847384f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:45:47.765934 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.765897    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98496502-c9bc-49e2-8ce1-b9be847384f8" (UID: "98496502-c9bc-49e2-8ce1-b9be847384f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:45:47.766034 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.765944    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf" (OuterVolumeSpecName: "kube-api-access-d9fbf") pod "81516825-8bda-419c-a736-266b71b67ccf" (UID: "81516825-8bda-419c-a736-266b71b67ccf"). InnerVolumeSpecName "kube-api-access-d9fbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:45:47.766081 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.766070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj" (OuterVolumeSpecName: "kube-api-access-5gtqj") pod "98496502-c9bc-49e2-8ce1-b9be847384f8" (UID: "98496502-c9bc-49e2-8ce1-b9be847384f8"). InnerVolumeSpecName "kube-api-access-5gtqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865433    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865459    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98496502-c9bc-49e2-8ce1-b9be847384f8-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865468    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98496502-c9bc-49e2-8ce1-b9be847384f8-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865491    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fbf\" (UniqueName: \"kubernetes.io/projected/81516825-8bda-419c-a736-266b71b67ccf-kube-api-access-d9fbf\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865500    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gtqj\" (UniqueName: \"kubernetes.io/projected/98496502-c9bc-49e2-8ce1-b9be847384f8-kube-api-access-5gtqj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:47.865511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:47.865509    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81516825-8bda-419c-a736-266b71b67ccf-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:45:48.048082 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.048046    2567 generic.go:356] "Generic (PLEG): container finished" podID="98496502-c9bc-49e2-8ce1-b9be847384f8" containerID="5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682" exitCode=0
Sep 18 06:45:48.048269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.048091    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"
Sep 18 06:45:48.048269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.048088    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" event={"ID":"98496502-c9bc-49e2-8ce1-b9be847384f8","Type":"ContainerDied","Data":"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"}
Sep 18 06:45:48.048269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.048230    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw" event={"ID":"98496502-c9bc-49e2-8ce1-b9be847384f8","Type":"ContainerDied","Data":"ddcc5c1a3fbe8ceb3769f45378aa17dd7651bdfee98ba01dfeafce84d3b72d5f"}
Sep 18 06:45:48.048269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.048260    2567 scope.go:117] "RemoveContainer" containerID="5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"
Sep 18 06:45:48.049575 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.049554    2567 generic.go:356] "Generic (PLEG): container finished" podID="81516825-8bda-419c-a736-266b71b67ccf" containerID="59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b" exitCode=0
Sep 18 06:45:48.049658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.049609    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"
Sep 18 06:45:48.049658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.049615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" event={"ID":"81516825-8bda-419c-a736-266b71b67ccf","Type":"ContainerDied","Data":"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"}
Sep 18 06:45:48.049658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.049651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv" event={"ID":"81516825-8bda-419c-a736-266b71b67ccf","Type":"ContainerDied","Data":"e9cfa70e88b83f00398fbea686dfad562759e9ca1091c327506d1b787d0b76ce"}
Sep 18 06:45:48.057648 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.057629    2567 scope.go:117] "RemoveContainer" containerID="5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"
Sep 18 06:45:48.057977 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:45:48.057954    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682\": container with ID starting with 5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682 not found: ID does not exist" containerID="5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"
Sep 18 06:45:48.058040 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.057990    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682"} err="failed to get container status \"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682\": rpc error: code = NotFound desc = could not find container \"5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682\": container with ID starting with 5a57e5f30443d58f4ce6acf7cc0634150590a67bec63417b9140a74d00f1c682 not found: ID does not exist"
Sep 18 06:45:48.058040 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.058013    2567 scope.go:117] "RemoveContainer" containerID="59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"
Sep 18 06:45:48.066000 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.065984    2567 scope.go:117] "RemoveContainer" containerID="59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"
Sep 18 06:45:48.066245 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:45:48.066227    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b\": container with ID starting with 59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b not found: ID does not exist" containerID="59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"
Sep 18 06:45:48.066307 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.066251    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b"} err="failed to get container status \"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b\": rpc error: code = NotFound desc = could not find container \"59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b\": container with ID starting with 59708dd4b68fdcafe15512ca5304099bc64e7cf978a4be17ce37ab30c9c1233b not found: ID does not exist"
Sep 18 06:45:48.075149 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.075111    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:48.078167 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.078126    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d887938f-9c9698b7b-b8zlw"]
Sep 18 06:45:48.086777 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.086755    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:48.091341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.091320    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d887938f-644585b75c-h5xkv"]
Sep 18 06:45:48.910793 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.910762    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81516825-8bda-419c-a736-266b71b67ccf" path="/var/lib/kubelet/pods/81516825-8bda-419c-a736-266b71b67ccf/volumes"
Sep 18 06:45:48.911231 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:45:48.911215    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98496502-c9bc-49e2-8ce1-b9be847384f8" path="/var/lib/kubelet/pods/98496502-c9bc-49e2-8ce1-b9be847384f8/volumes"
Sep 18 06:46:09.875847 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.875812    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:09.876340 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876256    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98496502-c9bc-49e2-8ce1-b9be847384f8" containerName="maas-api"
Sep 18 06:46:09.876340 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876268    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="98496502-c9bc-49e2-8ce1-b9be847384f8" containerName="maas-api"
Sep 18 06:46:09.876340 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876280    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81516825-8bda-419c-a736-266b71b67ccf" containerName="payload-processing"
Sep 18 06:46:09.876340 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876286    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81516825-8bda-419c-a736-266b71b67ccf" containerName="payload-processing"
Sep 18 06:46:09.876484 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876361    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81516825-8bda-419c-a736-266b71b67ccf" containerName="payload-processing"
Sep 18 06:46:09.876484 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.876374    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="98496502-c9bc-49e2-8ce1-b9be847384f8" containerName="maas-api"
Sep 18 06:46:09.880866 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.880851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.883386 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.883366    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-14ee59\"/\"default-dockercfg-sl5wq\""
Sep 18 06:46:09.883505 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.883388    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-14ee59\"/\"openshift-service-ca.crt\""
Sep 18 06:46:09.883505 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.883398    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-14ee59\"/\"kube-root-ca.crt\""
Sep 18 06:46:09.884568 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.884551    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-14ee59\"/\"explicit-model-14ee59-kserve-self-signed-certs\""
Sep 18 06:46:09.888161 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.888115    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:09.979046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979011    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.979240 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.979240 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.979240 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnb8l\" (UniqueName: \"kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.979359 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:09.979359 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:09.979289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080516 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080534    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnb8l\" (UniqueName: \"kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080618    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.080678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.081027 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.080999    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.081027 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.081013    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.081220 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.081086    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.082801 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.082771    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.083064 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.083046    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.088316 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.088296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnb8l\" (UniqueName: \"kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l\") pod \"explicit-model-14ee59-kserve-56559846cf-wxhfx\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") " pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.191637 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.191553    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:10.462904 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:10.462882    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:10.465523 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:46:10.465494    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75fa2712_433d_4cf6_962e_d52c0e462f6e.slice/crio-536b5bc2cd57b6386fc93178b8f9631457e305867aca6f6a08d3d20067e6029c WatchSource:0}: Error finding container 536b5bc2cd57b6386fc93178b8f9631457e305867aca6f6a08d3d20067e6029c: Status 404 returned error can't find the container with id 536b5bc2cd57b6386fc93178b8f9631457e305867aca6f6a08d3d20067e6029c
Sep 18 06:46:11.167318 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:11.167278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" event={"ID":"75fa2712-433d-4cf6-962e-d52c0e462f6e","Type":"ContainerStarted","Data":"ae4105d4737731c6cf9c90ecccbd7687f65c732febf06a4466d45c4aa0b6f25a"}
Sep 18 06:46:11.167761 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:11.167328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" event={"ID":"75fa2712-433d-4cf6-962e-d52c0e462f6e","Type":"ContainerStarted","Data":"536b5bc2cd57b6386fc93178b8f9631457e305867aca6f6a08d3d20067e6029c"}
Sep 18 06:46:20.192670 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:20.192635    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:20.193063 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:20.192711    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:20.205330 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:20.205307    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:20.222194 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:20.222172    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:20.222194 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:20.222163    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" podStartSLOduration=11.222146844 podStartE2EDuration="11.222146844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:11.183613124 +0000 UTC m=+1450.766998779" watchObservedRunningTime="2026-09-18 06:46:20.222146844 +0000 UTC m=+1459.805532490"
Sep 18 06:46:29.589008 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.588960    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:46:29.593664 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.593639    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.598012 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.597742    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"default-dockercfg-f7j4n\""
Sep 18 06:46:29.598012 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.597793    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"openshift-service-ca.crt\""
Sep 18 06:46:29.598012 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.597832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"e2e-conflict-b-24f27817-kserve-self-signed-certs\""
Sep 18 06:46:29.598500 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.598479    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"kube-root-ca.crt\""
Sep 18 06:46:29.600721 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.600695    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:46:29.670954 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.670920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.670954 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.670960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.671241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.670996    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.671241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.671058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.671241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.671171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.671241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.671213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5ss\" (UniqueName: \"kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.772661 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.772849 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.772849 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.772849 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5ss\" (UniqueName: \"kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.773030 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772881    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.773030 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.772909    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.773416 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.773385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.773674 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.773650    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.773902 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.773883    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.775616 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.775589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.775711 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.775633    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.780348 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.780320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5ss\" (UniqueName: \"kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss\") pod \"e2e-conflict-b-24f27817-kserve-596b789bdd-msblh\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") " pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:29.905759 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:29.905675    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:30.183731 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:30.183642    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:46:30.186154 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:46:30.186103    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e9fec69_95a9_4f1f_a83d_6982f5b76829.slice/crio-51fbeb9446f3963deb63bc52e27865d72c8cc02b9a9b7a5042556e8ddc08083c WatchSource:0}: Error finding container 51fbeb9446f3963deb63bc52e27865d72c8cc02b9a9b7a5042556e8ddc08083c: Status 404 returned error can't find the container with id 51fbeb9446f3963deb63bc52e27865d72c8cc02b9a9b7a5042556e8ddc08083c
Sep 18 06:46:30.277761 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:30.277723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" event={"ID":"0e9fec69-95a9-4f1f-a83d-6982f5b76829","Type":"ContainerStarted","Data":"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"}
Sep 18 06:46:30.277899 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:30.277774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" event={"ID":"0e9fec69-95a9-4f1f-a83d-6982f5b76829","Type":"ContainerStarted","Data":"51fbeb9446f3963deb63bc52e27865d72c8cc02b9a9b7a5042556e8ddc08083c"}
Sep 18 06:46:30.294650 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:30.294597    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" podStartSLOduration=1.2945812509999999 podStartE2EDuration="1.294581251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:30.292754055 +0000 UTC m=+1469.876139708" watchObservedRunningTime="2026-09-18 06:46:30.294581251 +0000 UTC m=+1469.877966905"
Sep 18 06:46:33.162668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:33.162636    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:39.906703 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:39.906661    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:39.907108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:39.906728    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:39.920828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:39.920800    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:40.342406 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:40.342374    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:46:47.549966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.549926    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:46:47.550798 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.550740    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" podUID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" containerName="maas-api" containerID="cri-o://22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807" gracePeriod=30
Sep 18 06:46:47.817168 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.817119    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:46:47.819171 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.819148    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:46:47.821206 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.821186    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:47.823595 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.823566    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-87d618-serving-cert\""
Sep 18 06:46:47.830087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.829200    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:46:47.958215 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958177    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca\") pod \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") "
Sep 18 06:46:47.958372 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958311    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgvj6\" (UniqueName: \"kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6\") pod \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") "
Sep 18 06:46:47.958372 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958356    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") pod \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") "
Sep 18 06:46:47.958464 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958410    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs\") pod \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\" (UID: \"582212b8-cc3d-4de5-b5c2-ad509ffd0bde\") "
Sep 18 06:46:47.958648 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958576    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "582212b8-cc3d-4de5-b5c2-ad509ffd0bde" (UID: "582212b8-cc3d-4de5-b5c2-ad509ffd0bde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:46:47.958648 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrvp\" (UniqueName: \"kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:47.958828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958783    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:47.958885 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958849    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:47.958942 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.958899    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:47.959040 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.959017    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:47.960992 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.960964    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "582212b8-cc3d-4de5-b5c2-ad509ffd0bde" (UID: "582212b8-cc3d-4de5-b5c2-ad509ffd0bde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:46:47.961111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.960989    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "582212b8-cc3d-4de5-b5c2-ad509ffd0bde" (UID: "582212b8-cc3d-4de5-b5c2-ad509ffd0bde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:46:47.961111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:47.961031    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6" (OuterVolumeSpecName: "kube-api-access-wgvj6") pod "582212b8-cc3d-4de5-b5c2-ad509ffd0bde" (UID: "582212b8-cc3d-4de5-b5c2-ad509ffd0bde"). InnerVolumeSpecName "kube-api-access-wgvj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:46:48.059537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrvp\" (UniqueName: \"kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.059698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059611    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.059698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.059698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.059855 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059794    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvj6\" (UniqueName: \"kubernetes.io/projected/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-kube-api-access-wgvj6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.059855 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059820    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.059855 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.059836    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/582212b8-cc3d-4de5-b5c2-ad509ffd0bde-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.059855 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:46:48.059821    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-serving-cert: secret "maas-api-e2e-shared-b-w2-87d618-serving-cert" not found
Sep 18 06:46:48.060063 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:46:48.059902    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls podName:633cece7-7c06-4ef4-84eb-9a68ff8cf796 nodeName:}" failed. No retries permitted until 2026-09-18 06:46:48.559883169 +0000 UTC m=+1488.143268819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls") pod "maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" (UID: "633cece7-7c06-4ef4-84eb-9a68ff8cf796") : secret "maas-api-e2e-shared-b-w2-87d618-serving-cert" not found
Sep 18 06:46:48.060458 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.060434    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.062228 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.062203    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.069851 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.069829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrvp\" (UniqueName: \"kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.183100 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.183057    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" podUID="75fa2712-433d-4cf6-962e-d52c0e462f6e" containerName="main" containerID="cri-o://ae4105d4737731c6cf9c90ecccbd7687f65c732febf06a4466d45c4aa0b6f25a" gracePeriod=45
Sep 18 06:46:48.377941 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.377905    2567 generic.go:356] "Generic (PLEG): container finished" podID="75fa2712-433d-4cf6-962e-d52c0e462f6e" containerID="ae4105d4737731c6cf9c90ecccbd7687f65c732febf06a4466d45c4aa0b6f25a" exitCode=0
Sep 18 06:46:48.378092 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.377975    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" event={"ID":"75fa2712-433d-4cf6-962e-d52c0e462f6e","Type":"ContainerDied","Data":"ae4105d4737731c6cf9c90ecccbd7687f65c732febf06a4466d45c4aa0b6f25a"}
Sep 18 06:46:48.379571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.379508    2567 generic.go:356] "Generic (PLEG): container finished" podID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" containerID="22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807" exitCode=0
Sep 18 06:46:48.379571 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.379546    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" event={"ID":"582212b8-cc3d-4de5-b5c2-ad509ffd0bde","Type":"ContainerDied","Data":"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"}
Sep 18 06:46:48.379737 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.379612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb" event={"ID":"582212b8-cc3d-4de5-b5c2-ad509ffd0bde","Type":"ContainerDied","Data":"1ac2c28ad6ab1e142ddf3ed8175eb6b4e3389792eb355d52992b9bd1e81f4597"}
Sep 18 06:46:48.379737 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.379647    2567 scope.go:117] "RemoveContainer" containerID="22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"
Sep 18 06:46:48.379737 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.379715    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"
Sep 18 06:46:48.393108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.393088    2567 scope.go:117] "RemoveContainer" containerID="22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"
Sep 18 06:46:48.393480 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:46:48.393458    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807\": container with ID starting with 22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807 not found: ID does not exist" containerID="22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"
Sep 18 06:46:48.393583 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.393485    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807"} err="failed to get container status \"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807\": rpc error: code = NotFound desc = could not find container \"22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807\": container with ID starting with 22d0549d477644777304088b12d554b2517cc33d8e94e0ecc25db733ea124807 not found: ID does not exist"
Sep 18 06:46:48.409966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.409939    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:46:48.415908 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.415883    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-14ee59-f599fd4f9-hngjb"]
Sep 18 06:46:48.430489 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.430468    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:48.564825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.564792    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.564871    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.564908    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.564932    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnb8l\" (UniqueName: \"kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.564984    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache\") pod \"75fa2712-433d-4cf6-962e-d52c0e462f6e\" (UID: \"75fa2712-433d-4cf6-962e-d52c0e462f6e\") "
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565103    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home" (OuterVolumeSpecName: "home") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:46:48.565286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.565642 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565442    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.565642 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565479    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:46:48.565642 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.565524    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:46:48.567625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.567594    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l" (OuterVolumeSpecName: "kube-api-access-dnb8l") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "kube-api-access-dnb8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:46:48.567755 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.567686    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm" (OuterVolumeSpecName: "dshm") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:46:48.567755 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.567726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.567755 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.567732    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75fa2712-433d-4cf6-962e-d52c0e462f6e" (UID: "75fa2712-433d-4cf6-962e-d52c0e462f6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:46:48.666715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.666635    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.666715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.666662    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnb8l\" (UniqueName: \"kubernetes.io/projected/75fa2712-433d-4cf6-962e-d52c0e462f6e-kube-api-access-dnb8l\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.666715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.666673    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.666715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.666681    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75fa2712-433d-4cf6-962e-d52c0e462f6e-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.666715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.666689    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75fa2712-433d-4cf6-962e-d52c0e462f6e-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:46:48.736704 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.736667    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:48.910753 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:48.910723    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" path="/var/lib/kubelet/pods/582212b8-cc3d-4de5-b5c2-ad509ffd0bde/volumes"
Sep 18 06:46:49.009152 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.009106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:46:49.010969 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:46:49.010941    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633cece7_7c06_4ef4_84eb_9a68ff8cf796.slice/crio-de78cfa5a2674ac839b8a18035714d607d4fbf55df3c10b28ac91d51b9ddc49b WatchSource:0}: Error finding container de78cfa5a2674ac839b8a18035714d607d4fbf55df3c10b28ac91d51b9ddc49b: Status 404 returned error can't find the container with id de78cfa5a2674ac839b8a18035714d607d4fbf55df3c10b28ac91d51b9ddc49b
Sep 18 06:46:49.386977 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.386936    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" event={"ID":"633cece7-7c06-4ef4-84eb-9a68ff8cf796","Type":"ContainerStarted","Data":"de78cfa5a2674ac839b8a18035714d607d4fbf55df3c10b28ac91d51b9ddc49b"}
Sep 18 06:46:49.388441 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.388411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx" event={"ID":"75fa2712-433d-4cf6-962e-d52c0e462f6e","Type":"ContainerDied","Data":"536b5bc2cd57b6386fc93178b8f9631457e305867aca6f6a08d3d20067e6029c"}
Sep 18 06:46:49.388575 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.388459    2567 scope.go:117] "RemoveContainer" containerID="ae4105d4737731c6cf9c90ecccbd7687f65c732febf06a4466d45c4aa0b6f25a"
Sep 18 06:46:49.388575 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.388458    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"
Sep 18 06:46:49.403401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.403379    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:49.409470 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:49.409450    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-14ee59/explicit-model-14ee59-kserve-56559846cf-wxhfx"]
Sep 18 06:46:50.397491 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:50.397457    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" event={"ID":"633cece7-7c06-4ef4-84eb-9a68ff8cf796","Type":"ContainerStarted","Data":"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"}
Sep 18 06:46:50.397897 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:50.397573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:46:50.416019 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:50.415970    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" podStartSLOduration=3.020118324 podStartE2EDuration="3.415954329s" totalImagesPullingTime="395.836005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:46:50.413165176 +0000 UTC m=+1489.996550833" watchObservedRunningTime="2026-09-18 06:46:50.415954329 +0000 UTC m=+1489.999339981"
Sep 18 06:46:50.910903 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:50.910869    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75fa2712-433d-4cf6-962e-d52c0e462f6e" path="/var/lib/kubelet/pods/75fa2712-433d-4cf6-962e-d52c0e462f6e/volumes"
Sep 18 06:46:56.408384 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:46:56.408354    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:47:00.924613 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:00.924580    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:47:00.928440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:00.928419    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:47:08.581078 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581041    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:47:08.581526 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581511    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75fa2712-433d-4cf6-962e-d52c0e462f6e" containerName="main"
Sep 18 06:47:08.581572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581529    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="75fa2712-433d-4cf6-962e-d52c0e462f6e" containerName="main"
Sep 18 06:47:08.581572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581544    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" containerName="maas-api"
Sep 18 06:47:08.581572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581551    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" containerName="maas-api"
Sep 18 06:47:08.581671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581617    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="582212b8-cc3d-4de5-b5c2-ad509ffd0bde" containerName="maas-api"
Sep 18 06:47:08.581671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.581625    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="75fa2712-433d-4cf6-962e-d52c0e462f6e" containerName="main"
Sep 18 06:47:08.584441 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.584425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.586868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.586848    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-946609\"/\"distinct-w0-946609-kserve-self-signed-certs\""
Sep 18 06:47:08.591128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.591100    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:47:08.665390 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.665548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2jw\" (UniqueName: \"kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.665548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.665548 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.665697 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665559    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.665697 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.665654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767013 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.766971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767013 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2jw\" (UniqueName: \"kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767283 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.767926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.768051 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.767950    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.768220 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.768193    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.770586 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.770560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.774210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.773154    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.777067 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.777043    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2jw\" (UniqueName: \"kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw\") pod \"distinct-w0-946609-kserve-7d7fbf4c49-kqv47\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") " pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:08.893769 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:08.893689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:09.163369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:09.163344    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:47:09.165467 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:47:09.165434    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdd08d85_617a_4c11_861a_16d92183cf4a.slice/crio-deaf6bcda0f71bdb36a13a477649f1e18c75abbc1ece2b693cf8a7fafe5a6f28 WatchSource:0}: Error finding container deaf6bcda0f71bdb36a13a477649f1e18c75abbc1ece2b693cf8a7fafe5a6f28: Status 404 returned error can't find the container with id deaf6bcda0f71bdb36a13a477649f1e18c75abbc1ece2b693cf8a7fafe5a6f28
Sep 18 06:47:09.502701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:09.502665    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" event={"ID":"fdd08d85-617a-4c11-861a-16d92183cf4a","Type":"ContainerStarted","Data":"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"}
Sep 18 06:47:09.502875 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:09.502712    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" event={"ID":"fdd08d85-617a-4c11-861a-16d92183cf4a","Type":"ContainerStarted","Data":"deaf6bcda0f71bdb36a13a477649f1e18c75abbc1ece2b693cf8a7fafe5a6f28"}
Sep 18 06:47:09.519233 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:09.519188    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" podStartSLOduration=1.5191736850000002 podStartE2EDuration="1.519173685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:09.517745874 +0000 UTC m=+1509.101131528" watchObservedRunningTime="2026-09-18 06:47:09.519173685 +0000 UTC m=+1509.102559355"
Sep 18 06:47:12.777301 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:12.777267    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:47:18.893888 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:18.893854    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:18.894373 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:18.893909    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:18.910418 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:18.910394    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:19.564084 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:19.564057    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:47:27.796826 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:27.796764    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" podUID="e9de02f9-24cf-4045-b81b-aec16e797401" containerName="main" containerID="cri-o://c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb" gracePeriod=45
Sep 18 06:47:28.050991 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.050922    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:47:28.152842 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.152811    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.152855    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjnl4\" (UniqueName: \"kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.152906    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.152938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.152959    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153029 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153009    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home\") pod \"e9de02f9-24cf-4045-b81b-aec16e797401\" (UID: \"e9de02f9-24cf-4045-b81b-aec16e797401\") "
Sep 18 06:47:28.153350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153278    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:47:28.153350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153293    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:47:28.153350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153337    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home" (OuterVolumeSpecName: "home") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:47:28.153492 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153421    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.153492 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.153443    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.155122 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.155093    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:47:28.155640 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.155616    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm" (OuterVolumeSpecName: "dshm") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:47:28.155740 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.155642    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4" (OuterVolumeSpecName: "kube-api-access-cjnl4") pod "e9de02f9-24cf-4045-b81b-aec16e797401" (UID: "e9de02f9-24cf-4045-b81b-aec16e797401"). InnerVolumeSpecName "kube-api-access-cjnl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:47:28.254341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.254301    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.254341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.254334    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9de02f9-24cf-4045-b81b-aec16e797401-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.254341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.254345    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjnl4\" (UniqueName: \"kubernetes.io/projected/e9de02f9-24cf-4045-b81b-aec16e797401-kube-api-access-cjnl4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.254585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.254354    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9de02f9-24cf-4045-b81b-aec16e797401-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:28.598353 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.598317    2567 generic.go:356] "Generic (PLEG): container finished" podID="e9de02f9-24cf-4045-b81b-aec16e797401" containerID="c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb" exitCode=0
Sep 18 06:47:28.598532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.598364    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"
Sep 18 06:47:28.598532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.598383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" event={"ID":"e9de02f9-24cf-4045-b81b-aec16e797401","Type":"ContainerDied","Data":"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"}
Sep 18 06:47:28.598532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.598416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk" event={"ID":"e9de02f9-24cf-4045-b81b-aec16e797401","Type":"ContainerDied","Data":"4be07f8eadfe1fa62681a2619e5717b5925b051843f54470a97218c0b02c2764"}
Sep 18 06:47:28.598532 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.598435    2567 scope.go:117] "RemoveContainer" containerID="c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"
Sep 18 06:47:28.608111 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.608091    2567 scope.go:117] "RemoveContainer" containerID="c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"
Sep 18 06:47:28.608413 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:28.608394    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb\": container with ID starting with c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb not found: ID does not exist" containerID="c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"
Sep 18 06:47:28.608485 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.608420    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb"} err="failed to get container status \"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb\": rpc error: code = NotFound desc = could not find container \"c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb\": container with ID starting with c523e2c579a1ec770e86c71507b11c53036022fcdc415dd68e7ef3e4e4543efb not found: ID does not exist"
Sep 18 06:47:28.619100 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.619077    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:47:28.622859 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.622838    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7c4f4b/test-model-7c4f4b-kserve-58ff78f9b9-8gwzk"]
Sep 18 06:47:28.910494 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:28.910398    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9de02f9-24cf-4045-b81b-aec16e797401" path="/var/lib/kubelet/pods/e9de02f9-24cf-4045-b81b-aec16e797401/volumes"
Sep 18 06:47:30.304515 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.304480    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:47:30.304992 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.304915    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9de02f9-24cf-4045-b81b-aec16e797401" containerName="main"
Sep 18 06:47:30.304992 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.304926    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9de02f9-24cf-4045-b81b-aec16e797401" containerName="main"
Sep 18 06:47:30.305119 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.305047    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9de02f9-24cf-4045-b81b-aec16e797401" containerName="main"
Sep 18 06:47:30.309484 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.309462    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.311698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.311677    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-93683709-serving-cert\""
Sep 18 06:47:30.317003 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.316982    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:47:30.372011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.371984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.372011 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.372014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.372265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.372058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.372265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.372191    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjnh\" (UniqueName: \"kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.473607 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.473568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.473786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.473617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjnh\" (UniqueName: \"kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.473786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.473693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.473786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.473714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.473953 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:30.473842    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-93683709-serving-cert: secret "maas-api-e2e-mt-93683709-serving-cert" not found
Sep 18 06:47:30.473953 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:30.473917    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls podName:a4887f55-43c8-4423-a133-4c339192ea35 nodeName:}" failed. No retries permitted until 2026-09-18 06:47:30.973896201 +0000 UTC m=+1530.557281847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls") pod "maas-api-e2e-mt-93683709-68fddddbb8-rglbc" (UID: "a4887f55-43c8-4423-a133-4c339192ea35") : secret "maas-api-e2e-mt-93683709-serving-cert" not found
Sep 18 06:47:30.474245 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.474222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.476342 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.476325    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.481537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.481513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjnh\" (UniqueName: \"kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.884555 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.884519    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:47:30.888345 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.888329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:30.890983 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.890958    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-93683709\""
Sep 18 06:47:30.891375 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.891358    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-93683709-dockercfg-chh5h\""
Sep 18 06:47:30.899496 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.899471    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:47:30.911684 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.911661    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:47:30.919198 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.919175    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:47:30.919316 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.919260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:30.978982 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.978955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2rc\" (UniqueName: \"kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:30.979132 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.978994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:30.979132 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.979088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:30.979241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.979157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqktr\" (UniqueName: \"kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:30.979241 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.979196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:30.981280 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:30.981261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") pod \"maas-api-e2e-mt-93683709-68fddddbb8-rglbc\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:31.080602 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.080563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqktr\" (UniqueName: \"kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.080766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.080616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.080766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.080702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2rc\" (UniqueName: \"kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.080766 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.080737    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.081488 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.081466    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.081581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.081474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.088244 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.088223    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqktr\" (UniqueName: \"kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr\") pod \"payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.088388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.088366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2rc\" (UniqueName: \"kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc\") pod \"payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") " pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.200374 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.200340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.224252 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.224228    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:31.231885 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.231847    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.479338 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.479313    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:47:31.480804 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:47:31.480774    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6a2847_e32f_4c73_bc12_3b094c4ad920.slice/crio-ece6056d0534434c89b992307415f3cbca86803a9675eb870a54edf6924f9ebc WatchSource:0}: Error finding container ece6056d0534434c89b992307415f3cbca86803a9675eb870a54edf6924f9ebc: Status 404 returned error can't find the container with id ece6056d0534434c89b992307415f3cbca86803a9675eb870a54edf6924f9ebc
Sep 18 06:47:31.518868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.518733    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:47:31.522679 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.522655    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:47:31.526552 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:47:31.526517    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4887f55_43c8_4423_a133_4c339192ea35.slice/crio-273c8ce2de3ac2d508cb0b622eb3aa625a2ee95a8b24ea624fd26bea62592af9 WatchSource:0}: Error finding container 273c8ce2de3ac2d508cb0b622eb3aa625a2ee95a8b24ea624fd26bea62592af9: Status 404 returned error can't find the container with id 273c8ce2de3ac2d508cb0b622eb3aa625a2ee95a8b24ea624fd26bea62592af9
Sep 18 06:47:31.526768 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:47:31.526738    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69ea603_0964_4a1b_a3f6_adec1ee3fdd1.slice/crio-35af757a7b426c6dc4b2fe116527e744becdb44ad23707f391daa3cfa8d97add WatchSource:0}: Error finding container 35af757a7b426c6dc4b2fe116527e744becdb44ad23707f391daa3cfa8d97add: Status 404 returned error can't find the container with id 35af757a7b426c6dc4b2fe116527e744becdb44ad23707f391daa3cfa8d97add
Sep 18 06:47:31.616379 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.616338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" event={"ID":"a4887f55-43c8-4423-a133-4c339192ea35","Type":"ContainerStarted","Data":"273c8ce2de3ac2d508cb0b622eb3aa625a2ee95a8b24ea624fd26bea62592af9"}
Sep 18 06:47:31.618280 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.618090    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" event={"ID":"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1","Type":"ContainerStarted","Data":"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"}
Sep 18 06:47:31.618430 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.618313    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" event={"ID":"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1","Type":"ContainerStarted","Data":"35af757a7b426c6dc4b2fe116527e744becdb44ad23707f391daa3cfa8d97add"}
Sep 18 06:47:31.618430 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.618345    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:31.620162 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.620124    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" event={"ID":"cb6a2847-e32f-4c73-bc12-3b094c4ad920","Type":"ContainerStarted","Data":"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"}
Sep 18 06:47:31.620265 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.620178    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" event={"ID":"cb6a2847-e32f-4c73-bc12-3b094c4ad920","Type":"ContainerStarted","Data":"ece6056d0534434c89b992307415f3cbca86803a9675eb870a54edf6924f9ebc"}
Sep 18 06:47:31.620327 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.620267    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:31.633773 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.633724    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" podStartSLOduration=1.633710138 podStartE2EDuration="1.633710138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:31.632966562 +0000 UTC m=+1531.216352214" watchObservedRunningTime="2026-09-18 06:47:31.633710138 +0000 UTC m=+1531.217095791"
Sep 18 06:47:31.649114 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:31.649067    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" podStartSLOduration=1.6490529600000001 podStartE2EDuration="1.64905296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:31.647224174 +0000 UTC m=+1531.230609829" watchObservedRunningTime="2026-09-18 06:47:31.64905296 +0000 UTC m=+1531.232438612"
Sep 18 06:47:32.627746 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:32.627708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" event={"ID":"a4887f55-43c8-4423-a133-4c339192ea35","Type":"ContainerStarted","Data":"b25fd3c99e7d0e80e2d0d2717bc3ff9a336b2dcbbbac6cc5cfefbdc8cf1c44c4"}
Sep 18 06:47:32.628230 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:32.628082    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:32.646183 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:32.646099    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" podStartSLOduration=2.237592991 podStartE2EDuration="2.646077933s" totalImagesPullingTime="408.484942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:47:32.643331187 +0000 UTC m=+1532.226716841" watchObservedRunningTime="2026-09-18 06:47:32.646077933 +0000 UTC m=+1532.229463587"
Sep 18 06:47:34.750873 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.750841    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:47:34.751368 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.751042    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" podUID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" containerName="maas-api" containerID="cri-o://f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854" gracePeriod=30
Sep 18 06:47:34.793231 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.793192    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:47:34.793430 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.793397    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" podUID="13114a20-a24d-4576-b8b9-718db6509f4e" containerName="payload-processing" containerID="cri-o://dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685" gracePeriod=30
Sep 18 06:47:34.801556 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.801520    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:47:34.801786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:34.801754    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" podUID="115b42e3-d1bd-4608-8fcb-abb663825392" containerName="payload-pre-processing" containerID="cri-o://d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271" gracePeriod=30
Sep 18 06:47:35.101325 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.101296    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:47:35.104306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.104284    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:47:35.123332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.121688    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs\") pod \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") "
Sep 18 06:47:35.123332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.121745    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") pod \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") "
Sep 18 06:47:35.123332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.121795    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvmtf\" (UniqueName: \"kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf\") pod \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") "
Sep 18 06:47:35.123332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.121870    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca\") pod \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\" (UID: \"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1\") "
Sep 18 06:47:35.123332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.122517    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" (UID: "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:47:35.127567 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.127525    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" (UID: "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:47:35.127797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.127768    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" (UID: "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:47:35.130014 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.129981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf" (OuterVolumeSpecName: "kube-api-access-gvmtf") pod "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" (UID: "c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1"). InnerVolumeSpecName "kube-api-access-gvmtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:47:35.141377 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.141348    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:47:35.222768 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.222733    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdvwl\" (UniqueName: \"kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl\") pod \"115b42e3-d1bd-4608-8fcb-abb663825392\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") "
Sep 18 06:47:35.223026 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.223006    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume\") pod \"115b42e3-d1bd-4608-8fcb-abb663825392\" (UID: \"115b42e3-d1bd-4608-8fcb-abb663825392\") "
Sep 18 06:47:35.223212 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.223194    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rjd5\" (UniqueName: \"kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5\") pod \"13114a20-a24d-4576-b8b9-718db6509f4e\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") "
Sep 18 06:47:35.223399 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.223380    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume\") pod \"13114a20-a24d-4576-b8b9-718db6509f4e\" (UID: \"13114a20-a24d-4576-b8b9-718db6509f4e\") "
Sep 18 06:47:35.223995 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.223969    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.224127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.224114    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.224263 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.224247    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmtf\" (UniqueName: \"kubernetes.io/projected/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-kube-api-access-gvmtf\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.224365 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.224351    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.224657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.224627    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13114a20-a24d-4576-b8b9-718db6509f4e" (UID: "13114a20-a24d-4576-b8b9-718db6509f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:47:35.224749 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.224627    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "115b42e3-d1bd-4608-8fcb-abb663825392" (UID: "115b42e3-d1bd-4608-8fcb-abb663825392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:47:35.227323 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.227278    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5" (OuterVolumeSpecName: "kube-api-access-5rjd5") pod "13114a20-a24d-4576-b8b9-718db6509f4e" (UID: "13114a20-a24d-4576-b8b9-718db6509f4e"). InnerVolumeSpecName "kube-api-access-5rjd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:47:35.227723 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.227692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl" (OuterVolumeSpecName: "kube-api-access-bdvwl") pod "115b42e3-d1bd-4608-8fcb-abb663825392" (UID: "115b42e3-d1bd-4608-8fcb-abb663825392"). InnerVolumeSpecName "kube-api-access-bdvwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:47:35.325547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.325450    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvwl\" (UniqueName: \"kubernetes.io/projected/115b42e3-d1bd-4608-8fcb-abb663825392-kube-api-access-bdvwl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.325547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.325483    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115b42e3-d1bd-4608-8fcb-abb663825392-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.325547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.325497    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjd5\" (UniqueName: \"kubernetes.io/projected/13114a20-a24d-4576-b8b9-718db6509f4e-kube-api-access-5rjd5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.325547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.325509    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13114a20-a24d-4576-b8b9-718db6509f4e-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:35.646127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.646042    2567 generic.go:356] "Generic (PLEG): container finished" podID="115b42e3-d1bd-4608-8fcb-abb663825392" containerID="d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271" exitCode=0
Sep 18 06:47:35.646127 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.646109    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"
Sep 18 06:47:35.646358 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.646130    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" event={"ID":"115b42e3-d1bd-4608-8fcb-abb663825392","Type":"ContainerDied","Data":"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"}
Sep 18 06:47:35.646358 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.646254    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt" event={"ID":"115b42e3-d1bd-4608-8fcb-abb663825392","Type":"ContainerDied","Data":"9ade35df28f37073bd0bc0063a1e273a8f693c4ab5e69eb306abab888d6fc493"}
Sep 18 06:47:35.646358 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.646289    2567 scope.go:117] "RemoveContainer" containerID="d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"
Sep 18 06:47:35.647556 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.647525    2567 generic.go:356] "Generic (PLEG): container finished" podID="13114a20-a24d-4576-b8b9-718db6509f4e" containerID="dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685" exitCode=0
Sep 18 06:47:35.647680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.647557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" event={"ID":"13114a20-a24d-4576-b8b9-718db6509f4e","Type":"ContainerDied","Data":"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"}
Sep 18 06:47:35.647680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.647612    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9" event={"ID":"13114a20-a24d-4576-b8b9-718db6509f4e","Type":"ContainerDied","Data":"aa700ca974b6b15018a1daa9e2b546861a9a3f27adfd4273c25ba44e07d2a634"}
Sep 18 06:47:35.647680 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.647571    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"
Sep 18 06:47:35.649311 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.649285    2567 generic.go:356] "Generic (PLEG): container finished" podID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" containerID="f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854" exitCode=0
Sep 18 06:47:35.649434 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.649340    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" event={"ID":"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1","Type":"ContainerDied","Data":"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"}
Sep 18 06:47:35.649434 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.649348    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"
Sep 18 06:47:35.649434 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.649376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq" event={"ID":"c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1","Type":"ContainerDied","Data":"6b39e3b48f7b6022ee6d3a6c73049c4e82ba1efecaf85358877b776bad05e8de"}
Sep 18 06:47:35.656388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.656357    2567 scope.go:117] "RemoveContainer" containerID="d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"
Sep 18 06:47:35.656703 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:35.656673    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271\": container with ID starting with d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271 not found: ID does not exist" containerID="d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"
Sep 18 06:47:35.656797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.656712    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271"} err="failed to get container status \"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271\": rpc error: code = NotFound desc = could not find container \"d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271\": container with ID starting with d69c72a2491129556e10b59e3390d1f7e098d070adbb627f2c5f4c7636a52271 not found: ID does not exist"
Sep 18 06:47:35.656797 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.656734    2567 scope.go:117] "RemoveContainer" containerID="dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"
Sep 18 06:47:35.667020 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.666998    2567 scope.go:117] "RemoveContainer" containerID="dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"
Sep 18 06:47:35.668775 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:35.668747    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685\": container with ID starting with dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685 not found: ID does not exist" containerID="dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"
Sep 18 06:47:35.668989 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.668948    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685"} err="failed to get container status \"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685\": rpc error: code = NotFound desc = could not find container \"dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685\": container with ID starting with dffe6a8022a5aea00ab77899c94546448af8b53be1a3a839dd29f44548b54685 not found: ID does not exist"
Sep 18 06:47:35.669152 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.669113    2567 scope.go:117] "RemoveContainer" containerID="f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"
Sep 18 06:47:35.671175 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.671061    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:47:35.673558 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.673538    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7c4f4b-676bf77998-zd5nt"]
Sep 18 06:47:35.678950 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.678927    2567 scope.go:117] "RemoveContainer" containerID="f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"
Sep 18 06:47:35.679270 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:35.679250    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854\": container with ID starting with f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854 not found: ID does not exist" containerID="f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"
Sep 18 06:47:35.679344 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.679280    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854"} err="failed to get container status \"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854\": rpc error: code = NotFound desc = could not find container \"f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854\": container with ID starting with f7b34d06b185f054dbfb322cfbb91c8ddb293212d33b157ab713da5768d46854 not found: ID does not exist"
Sep 18 06:47:35.683181 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.683158    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:47:35.688728 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.688704    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7c4f4b-f9bdcbd65-w2rg9"]
Sep 18 06:47:35.700928 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.700897    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:47:35.702748 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:35.702726    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7c4f4b-6f54797649-f6jfq"]
Sep 18 06:47:36.910343 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:36.910307    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="115b42e3-d1bd-4608-8fcb-abb663825392" path="/var/lib/kubelet/pods/115b42e3-d1bd-4608-8fcb-abb663825392/volumes"
Sep 18 06:47:36.910756 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:36.910742    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13114a20-a24d-4576-b8b9-718db6509f4e" path="/var/lib/kubelet/pods/13114a20-a24d-4576-b8b9-718db6509f4e/volumes"
Sep 18 06:47:36.911161 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:36.911146    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" path="/var/lib/kubelet/pods/c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1/volumes"
Sep 18 06:47:38.639423 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:38.639393    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:47:42.629800 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:42.629767    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:47:42.630199 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:42.629851    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:47:52.354297 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.354262    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:47:52.354837 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.354536    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" podUID="e8377056-7c62-491f-b5f1-257d6a3af96f" containerName="payload-processing" containerID="cri-o://a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77" gracePeriod=30
Sep 18 06:47:52.616858 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.616801    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:47:52.705601 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.705563    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume\") pod \"e8377056-7c62-491f-b5f1-257d6a3af96f\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") "
Sep 18 06:47:52.705772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.705711    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-886pc\" (UniqueName: \"kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc\") pod \"e8377056-7c62-491f-b5f1-257d6a3af96f\" (UID: \"e8377056-7c62-491f-b5f1-257d6a3af96f\") "
Sep 18 06:47:52.705957 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.705934    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8377056-7c62-491f-b5f1-257d6a3af96f" (UID: "e8377056-7c62-491f-b5f1-257d6a3af96f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:47:52.706052 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.706030    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8377056-7c62-491f-b5f1-257d6a3af96f-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:52.707879 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.707855    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc" (OuterVolumeSpecName: "kube-api-access-886pc") pod "e8377056-7c62-491f-b5f1-257d6a3af96f" (UID: "e8377056-7c62-491f-b5f1-257d6a3af96f"). InnerVolumeSpecName "kube-api-access-886pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:47:52.751718 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.751677    2567 generic.go:356] "Generic (PLEG): container finished" podID="e8377056-7c62-491f-b5f1-257d6a3af96f" containerID="a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77" exitCode=0
Sep 18 06:47:52.751718 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.751715    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"
Sep 18 06:47:52.751937 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.751764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" event={"ID":"e8377056-7c62-491f-b5f1-257d6a3af96f","Type":"ContainerDied","Data":"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"}
Sep 18 06:47:52.751937 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.751869    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph" event={"ID":"e8377056-7c62-491f-b5f1-257d6a3af96f","Type":"ContainerDied","Data":"9975f81d5e5dbba5fee56c85ddf7062cd69c3f3afd6b5b4f6ce303055fb76b57"}
Sep 18 06:47:52.751937 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.751905    2567 scope.go:117] "RemoveContainer" containerID="a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"
Sep 18 06:47:52.760998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.760983    2567 scope.go:117] "RemoveContainer" containerID="a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"
Sep 18 06:47:52.761310 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:47:52.761291    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77\": container with ID starting with a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77 not found: ID does not exist" containerID="a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"
Sep 18 06:47:52.761387 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.761319    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77"} err="failed to get container status \"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77\": rpc error: code = NotFound desc = could not find container \"a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77\": container with ID starting with a66dfa98226aada05f4d223526c4a8ca9f4960df1bdee101ae9c282217d1bd77 not found: ID does not exist"
Sep 18 06:47:52.775631 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.775605    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:47:52.777641 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.777618    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-708b6f-74c477c5d9-hk8ph"]
Sep 18 06:47:52.806864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.806840    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886pc\" (UniqueName: \"kubernetes.io/projected/e8377056-7c62-491f-b5f1-257d6a3af96f-kube-api-access-886pc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:47:52.909983 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:47:52.909891    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8377056-7c62-491f-b5f1-257d6a3af96f" path="/var/lib/kubelet/pods/e8377056-7c62-491f-b5f1-257d6a3af96f/volumes"
Sep 18 06:48:12.185547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185503    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185901    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115b42e3-d1bd-4608-8fcb-abb663825392" containerName="payload-pre-processing"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185913    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="115b42e3-d1bd-4608-8fcb-abb663825392" containerName="payload-pre-processing"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185924    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13114a20-a24d-4576-b8b9-718db6509f4e" containerName="payload-processing"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185932    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13114a20-a24d-4576-b8b9-718db6509f4e" containerName="payload-processing"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185942    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" containerName="maas-api"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185948    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" containerName="maas-api"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185956    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8377056-7c62-491f-b5f1-257d6a3af96f" containerName="payload-processing"
Sep 18 06:48:12.186017 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.185961    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8377056-7c62-491f-b5f1-257d6a3af96f" containerName="payload-processing"
Sep 18 06:48:12.186482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.186046    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="115b42e3-d1bd-4608-8fcb-abb663825392" containerName="payload-pre-processing"
Sep 18 06:48:12.186482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.186053    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f8dcab-7b5b-45bf-871c-4a9036fd7ea1" containerName="maas-api"
Sep 18 06:48:12.186482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.186058    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8377056-7c62-491f-b5f1-257d6a3af96f" containerName="payload-processing"
Sep 18 06:48:12.186482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.186065    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13114a20-a24d-4576-b8b9-718db6509f4e" containerName="payload-processing"
Sep 18 06:48:12.189911 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.189888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.193757 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.193729    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2f5ffe\"/\"noten-model-2f5ffe-kserve-self-signed-certs\""
Sep 18 06:48:12.193867 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.193773    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2f5ffe\"/\"openshift-service-ca.crt\""
Sep 18 06:48:12.193867 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.193815    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2f5ffe\"/\"default-dockercfg-89wfg\""
Sep 18 06:48:12.194071 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.193747    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2f5ffe\"/\"kube-root-ca.crt\""
Sep 18 06:48:12.199437 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.199415    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:12.290609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290574    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.290609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.290810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.290810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290678    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.290810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.290810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.290720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l69v\" (UniqueName: \"kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.391967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.391936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.391973    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392221 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392221 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392221 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l69v\" (UniqueName: \"kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392396 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392376    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392473 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392441    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.392561 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.392539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.394313 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.394287    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.394692 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.394672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.400036 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.400012    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l69v\" (UniqueName: \"kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v\") pod \"noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") " pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.500895 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.500859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:12.775607 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.773011    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:12.857768 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.857736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" event={"ID":"ba38fdfd-51b6-4eb8-8f5d-e5233678c814","Type":"ContainerStarted","Data":"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"}
Sep 18 06:48:12.857914 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.857784    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" event={"ID":"ba38fdfd-51b6-4eb8-8f5d-e5233678c814","Type":"ContainerStarted","Data":"617e0e75231fc0f45276c92bb8c5720cb6ae65cd9075b099de1bf2a189e357a3"}
Sep 18 06:48:12.877850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:12.877799    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" podStartSLOduration=0.87777091 podStartE2EDuration="877.77091ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:48:12.875623542 +0000 UTC m=+1572.459009195" watchObservedRunningTime="2026-09-18 06:48:12.87777091 +0000 UTC m=+1572.461156584"
Sep 18 06:48:22.501104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:22.501063    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:22.501104 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:22.501116    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:22.513764 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:22.513740    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:22.917368 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:22.917292    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:36.471451 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:36.471409    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:48:38.008925 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:38.008884    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:42.416208 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.416167    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 06:48:42.419893 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.419871    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.422608 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.422584    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ec2077-serving-cert\""
Sep 18 06:48:42.431860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.431836    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 06:48:42.583939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.583906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.584116 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.583946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.584116 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.583970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5d8j\" (UniqueName: \"kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.584116 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.584067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.685473 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.685380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.685473 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.685431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.685473 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.685461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5d8j\" (UniqueName: \"kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.685740 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.685548    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.686257 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.686230    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.687926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.687900    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.687926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.687914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.692581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.692556    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5d8j\" (UniqueName: \"kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j\") pod \"maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.732930 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.732903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:42.993572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.993542    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 06:48:42.997603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.997582    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:42.999926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.999903    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ec2077\""
Sep 18 06:48:43.000013 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:42.999922    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ec2077-dockercfg-qkw88\""
Sep 18 06:48:43.008926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.008905    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 06:48:43.011698 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.011674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 06:48:43.013335 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:48:43.013306    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531fe294_1340_42f6_884e_ec15bf20d614.slice/crio-f135ad3c3fc09381c06b6f56f8ec4cc90dc1368148c6f911534e71def26fccac WatchSource:0}: Error finding container f135ad3c3fc09381c06b6f56f8ec4cc90dc1368148c6f911534e71def26fccac: Status 404 returned error can't find the container with id f135ad3c3fc09381c06b6f56f8ec4cc90dc1368148c6f911534e71def26fccac
Sep 18 06:48:43.014779 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.014756    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:48:43.025507 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.025483    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 06:48:43.029366 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.029348    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.037270 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.037246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 06:48:43.191093 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.191046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.191304 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.191179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpq7\" (UniqueName: \"kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.191355 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.191295    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfzc\" (UniqueName: \"kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.191355 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.191346    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.292715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.292632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpq7\" (UniqueName: \"kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.292882 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.292735    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfzc\" (UniqueName: \"kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.292882 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.292775    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.292882 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.292816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.293574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.293545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.293574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.293562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.300172 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.300149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfzc\" (UniqueName: \"kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc\") pod \"payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.300259 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.300186    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpq7\" (UniqueName: \"kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7\") pod \"payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.310016 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.310000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:43.340289 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.340260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:43.583507 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.583480    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 06:48:43.585308 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:48:43.585272    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25ef891_9846_447c_825f_501808accd4f.slice/crio-41d3abd84817c9684ca32fa7b34facf1af440bc6e22eab826251728994f1273c WatchSource:0}: Error finding container 41d3abd84817c9684ca32fa7b34facf1af440bc6e22eab826251728994f1273c: Status 404 returned error can't find the container with id 41d3abd84817c9684ca32fa7b34facf1af440bc6e22eab826251728994f1273c
Sep 18 06:48:43.628756 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:43.628732    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 06:48:43.638340 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:48:43.638312    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd430db3b_2a38_40fe_8dda_cce3fe84aa96.slice/crio-4de66fb6e75b8aa66869eb717c573b3bc87a2e11ff3e2f0eb2b6ae2e2f65a51e WatchSource:0}: Error finding container 4de66fb6e75b8aa66869eb717c573b3bc87a2e11ff3e2f0eb2b6ae2e2f65a51e: Status 404 returned error can't find the container with id 4de66fb6e75b8aa66869eb717c573b3bc87a2e11ff3e2f0eb2b6ae2e2f65a51e
Sep 18 06:48:44.018482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.018436    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" event={"ID":"d430db3b-2a38-40fe-8dda-cce3fe84aa96","Type":"ContainerStarted","Data":"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"}
Sep 18 06:48:44.018678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.018495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" event={"ID":"d430db3b-2a38-40fe-8dda-cce3fe84aa96","Type":"ContainerStarted","Data":"4de66fb6e75b8aa66869eb717c573b3bc87a2e11ff3e2f0eb2b6ae2e2f65a51e"}
Sep 18 06:48:44.018678 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.018525    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:48:44.020338 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.020305    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" event={"ID":"a25ef891-9846-447c-825f-501808accd4f","Type":"ContainerStarted","Data":"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"}
Sep 18 06:48:44.020493 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.020372    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:44.020493 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.020400    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" event={"ID":"a25ef891-9846-447c-825f-501808accd4f","Type":"ContainerStarted","Data":"41d3abd84817c9684ca32fa7b34facf1af440bc6e22eab826251728994f1273c"}
Sep 18 06:48:44.022008 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.021981    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" event={"ID":"531fe294-1340-42f6-884e-ec15bf20d614","Type":"ContainerStarted","Data":"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"}
Sep 18 06:48:44.022153 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.022020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" event={"ID":"531fe294-1340-42f6-884e-ec15bf20d614","Type":"ContainerStarted","Data":"f135ad3c3fc09381c06b6f56f8ec4cc90dc1368148c6f911534e71def26fccac"}
Sep 18 06:48:44.022153 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.022086    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:44.035007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.034962    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" podStartSLOduration=1.034947371 podStartE2EDuration="1.034947371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:48:44.033927052 +0000 UTC m=+1603.617312709" watchObservedRunningTime="2026-09-18 06:48:44.034947371 +0000 UTC m=+1603.618333024"
Sep 18 06:48:44.050537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.050488    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" podStartSLOduration=2.050470064 podStartE2EDuration="2.050470064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:48:44.047853056 +0000 UTC m=+1603.631238712" watchObservedRunningTime="2026-09-18 06:48:44.050470064 +0000 UTC m=+1603.633855717"
Sep 18 06:48:44.064709 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:44.064662    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" podStartSLOduration=1.6910831979999998 podStartE2EDuration="2.064651807s" totalImagesPullingTime="373.568609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:48:44.062705156 +0000 UTC m=+1603.646090810" watchObservedRunningTime="2026-09-18 06:48:44.064651807 +0000 UTC m=+1603.648037498"
Sep 18 06:48:50.033966 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:50.033938    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 06:48:51.491356 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.491288    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" podUID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" containerName="main" containerID="cri-o://0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6" gracePeriod=45
Sep 18 06:48:51.745335 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.745274    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:48:51.774458 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774430    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774614 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774483    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg5ss\" (UniqueName: \"kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774614 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774554    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774614 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774581    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774791 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774735    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774802    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home\") pod \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\" (UID: \"0e9fec69-95a9-4f1f-a83d-6982f5b76829\") "
Sep 18 06:48:51.774853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:51.774853 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.774819    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:51.775117 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.775092    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home" (OuterVolumeSpecName: "home") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:51.775255 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.775240    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:51.775327 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.775256    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:51.775327 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.775268    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:51.776767 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.776743    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss" (OuterVolumeSpecName: "kube-api-access-kg5ss") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "kube-api-access-kg5ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:48:51.777025 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.777004    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm" (OuterVolumeSpecName: "dshm") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:51.777098 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.777021    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e9fec69-95a9-4f1f-a83d-6982f5b76829" (UID: "0e9fec69-95a9-4f1f-a83d-6982f5b76829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:48:51.876397 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.876366    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9fec69-95a9-4f1f-a83d-6982f5b76829-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:51.876397 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.876390    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9fec69-95a9-4f1f-a83d-6982f5b76829-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:51.876397 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:51.876399    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5ss\" (UniqueName: \"kubernetes.io/projected/0e9fec69-95a9-4f1f-a83d-6982f5b76829-kube-api-access-kg5ss\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:52.068915 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.068828    2567 generic.go:356] "Generic (PLEG): container finished" podID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" containerID="0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6" exitCode=0
Sep 18 06:48:52.068915 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.068875    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"
Sep 18 06:48:52.069087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.068911    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" event={"ID":"0e9fec69-95a9-4f1f-a83d-6982f5b76829","Type":"ContainerDied","Data":"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"}
Sep 18 06:48:52.069087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.068955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh" event={"ID":"0e9fec69-95a9-4f1f-a83d-6982f5b76829","Type":"ContainerDied","Data":"51fbeb9446f3963deb63bc52e27865d72c8cc02b9a9b7a5042556e8ddc08083c"}
Sep 18 06:48:52.069087 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.068972    2567 scope.go:117] "RemoveContainer" containerID="0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"
Sep 18 06:48:52.078846 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.078828    2567 scope.go:117] "RemoveContainer" containerID="0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"
Sep 18 06:48:52.079103 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:48:52.079084    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6\": container with ID starting with 0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6 not found: ID does not exist" containerID="0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"
Sep 18 06:48:52.079188 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.079114    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6"} err="failed to get container status \"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6\": rpc error: code = NotFound desc = could not find container \"0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6\": container with ID starting with 0fdf59707ae0d48177464823269925a7ded73abee2be8856491b95d6c3f3dbd6 not found: ID does not exist"
Sep 18 06:48:52.090195 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.090163    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:48:52.095730 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.095708    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-654463/e2e-conflict-b-24f27817-kserve-596b789bdd-msblh"]
Sep 18 06:48:52.910431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:52.910394    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" path="/var/lib/kubelet/pods/0e9fec69-95a9-4f1f-a83d-6982f5b76829/volumes"
Sep 18 06:48:53.028544 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.028500    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" podUID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" containerName="main" containerID="cri-o://347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3" gracePeriod=45
Sep 18 06:48:53.272883 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.272861    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:53.388222 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388123    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388229    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388261    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388287    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l69v\" (UniqueName: \"kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache\") pod \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\" (UID: \"ba38fdfd-51b6-4eb8-8f5d-e5233678c814\") "
Sep 18 06:48:53.388654 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388521    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:53.388654 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388528    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home" (OuterVolumeSpecName: "home") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:53.388761 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388719    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:53.388810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388793    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:53.388864 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.388811    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:53.390405 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.390385    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm" (OuterVolumeSpecName: "dshm") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:48:53.390482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.390422    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v" (OuterVolumeSpecName: "kube-api-access-5l69v") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "kube-api-access-5l69v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:48:53.390482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.390428    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba38fdfd-51b6-4eb8-8f5d-e5233678c814" (UID: "ba38fdfd-51b6-4eb8-8f5d-e5233678c814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:48:53.489967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.489872    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:53.489967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.489903    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:53.489967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.489918    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l69v\" (UniqueName: \"kubernetes.io/projected/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-kube-api-access-5l69v\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:53.489967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:53.489930    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba38fdfd-51b6-4eb8-8f5d-e5233678c814-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:48:54.081324 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.081286    2567 generic.go:356] "Generic (PLEG): container finished" podID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" containerID="347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3" exitCode=0
Sep 18 06:48:54.081776 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.081335    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"
Sep 18 06:48:54.081776 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.081344    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" event={"ID":"ba38fdfd-51b6-4eb8-8f5d-e5233678c814","Type":"ContainerDied","Data":"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"}
Sep 18 06:48:54.081776 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.081383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv" event={"ID":"ba38fdfd-51b6-4eb8-8f5d-e5233678c814","Type":"ContainerDied","Data":"617e0e75231fc0f45276c92bb8c5720cb6ae65cd9075b099de1bf2a189e357a3"}
Sep 18 06:48:54.081776 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.081406    2567 scope.go:117] "RemoveContainer" containerID="347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"
Sep 18 06:48:54.091146 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.091115    2567 scope.go:117] "RemoveContainer" containerID="347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"
Sep 18 06:48:54.091435 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:48:54.091413    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3\": container with ID starting with 347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3 not found: ID does not exist" containerID="347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"
Sep 18 06:48:54.091504 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.091446    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3"} err="failed to get container status \"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3\": rpc error: code = NotFound desc = could not find container \"347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3\": container with ID starting with 347c60cecc71a8a4aa7c54cbb8219a7098fa60458b65159be8977e06be08b1b3 not found: ID does not exist"
Sep 18 06:48:54.102420 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.102373    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:54.107662 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.107638    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2f5ffe/noten-model-2f5ffe-kserve-65f5dbdbb7-8kqkv"]
Sep 18 06:48:54.910058 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:54.910021    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" path="/var/lib/kubelet/pods/ba38fdfd-51b6-4eb8-8f5d-e5233678c814/volumes"
Sep 18 06:48:55.030338 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:55.030304    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 06:48:55.030635 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:48:55.030614    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 06:49:16.161407 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.161371    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 06:49:16.161884 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.161866    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" containerName="main"
Sep 18 06:49:16.161936 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.161886    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" containerName="main"
Sep 18 06:49:16.161936 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.161899    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" containerName="main"
Sep 18 06:49:16.161936 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.161906    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" containerName="main"
Sep 18 06:49:16.162040 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.162008    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9fec69-95a9-4f1f-a83d-6982f5b76829" containerName="main"
Sep 18 06:49:16.162040 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.162026    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba38fdfd-51b6-4eb8-8f5d-e5233678c814" containerName="main"
Sep 18 06:49:16.165195 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.165177    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.167552 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.167531    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8fdd5b-serving-cert\""
Sep 18 06:49:16.176329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.176309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 06:49:16.317259 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.317218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswc9\" (UniqueName: \"kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.317447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.317332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.317501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.317451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.317501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.317487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.418228 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.418110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswc9\" (UniqueName: \"kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.418382 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.418238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.418382 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.418330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.418382 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.418361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.419036 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.419008    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.420745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.420712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.420840 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.420762    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.425878 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.425854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswc9\" (UniqueName: \"kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9\") pod \"maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.476269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.476237    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:16.749798 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:16.749773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 06:49:16.752206 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:49:16.752177    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e12cdfb_19c5_4057_b810_8f0559b73fc3.slice/crio-99fcad99204dd66011fc62fbf342de3f1447a4a65802c20d63bd68032b419920 WatchSource:0}: Error finding container 99fcad99204dd66011fc62fbf342de3f1447a4a65802c20d63bd68032b419920: Status 404 returned error can't find the container with id 99fcad99204dd66011fc62fbf342de3f1447a4a65802c20d63bd68032b419920
Sep 18 06:49:17.205440 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:17.205413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" event={"ID":"0e12cdfb-19c5-4057-b810-8f0559b73fc3","Type":"ContainerStarted","Data":"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"}
Sep 18 06:49:17.205789 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:17.205483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:49:17.205789 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:17.205514    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" event={"ID":"0e12cdfb-19c5-4057-b810-8f0559b73fc3","Type":"ContainerStarted","Data":"99fcad99204dd66011fc62fbf342de3f1447a4a65802c20d63bd68032b419920"}
Sep 18 06:49:17.225228 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:17.225179    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" podStartSLOduration=0.835585263 podStartE2EDuration="1.225163224s" totalImagesPullingTime="389.577961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:49:17.221993465 +0000 UTC m=+1636.805379120" watchObservedRunningTime="2026-09-18 06:49:17.225163224 +0000 UTC m=+1636.808548877"
Sep 18 06:49:23.215825 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:49:23.215789    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 06:50:08.686315 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.686272    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:50:08.690118 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.690092    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.692781 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.692753    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"kube-root-ca.crt\""
Sep 18 06:50:08.692894 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.692865    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"openshift-service-ca.crt\""
Sep 18 06:50:08.693810 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.693793    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"default-dockercfg-f7j4n\""
Sep 18 06:50:08.694292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.694265    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-654463\"/\"distinct-w4-654463-kserve-self-signed-certs\""
Sep 18 06:50:08.701236 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.701166    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:50:08.724998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.724962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.725157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.725019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.725157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.725100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrbv\" (UniqueName: \"kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.725157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.725150    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.725377 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.725197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.725377 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.725253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826264 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826436 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrbv\" (UniqueName: \"kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826708 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826708 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.826815 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.826753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.828622 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.828603    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.828964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.828946    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:08.833939 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:08.833917    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrbv\" (UniqueName: \"kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv\") pod \"distinct-w4-654463-kserve-5f44464c77-t6hnz\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") " pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:09.002296 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:09.002260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:09.270522 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:09.270492    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:50:09.272127 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:50:09.272103    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19cce629_d9ba_4379_9daf_c762bbce6c73.slice/crio-873060bbc8ff1254aa59fd5a892eb15b2d2fd3acd515a1626acce1a7ec786c11 WatchSource:0}: Error finding container 873060bbc8ff1254aa59fd5a892eb15b2d2fd3acd515a1626acce1a7ec786c11: Status 404 returned error can't find the container with id 873060bbc8ff1254aa59fd5a892eb15b2d2fd3acd515a1626acce1a7ec786c11
Sep 18 06:50:09.472369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:09.472333    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" event={"ID":"19cce629-d9ba-4379-9daf-c762bbce6c73","Type":"ContainerStarted","Data":"7eee84d7042415eaec3cb541a2a3f15cf564a3808f498a14525c952880649680"}
Sep 18 06:50:09.472369 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:09.472384    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" event={"ID":"19cce629-d9ba-4379-9daf-c762bbce6c73","Type":"ContainerStarted","Data":"873060bbc8ff1254aa59fd5a892eb15b2d2fd3acd515a1626acce1a7ec786c11"}
Sep 18 06:50:09.489926 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:09.489876    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" podStartSLOduration=1.489860938 podStartE2EDuration="1.489860938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:50:09.487940038 +0000 UTC m=+1689.071325692" watchObservedRunningTime="2026-09-18 06:50:09.489860938 +0000 UTC m=+1689.073246591"
Sep 18 06:50:19.002475 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:19.002438    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:19.002938 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:19.002491    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:19.015238 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:19.015213    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:19.538468 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:19.538441    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:50:35.456237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.456157    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:50:35.456760 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.456381    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" podUID="a4887f55-43c8-4423-a133-4c339192ea35" containerName="maas-api" containerID="cri-o://b25fd3c99e7d0e80e2d0d2717bc3ff9a336b2dcbbbac6cc5cfefbdc8cf1c44c4" gracePeriod=30
Sep 18 06:50:35.514758 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.514717    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:50:35.515891 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.515030    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" podUID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" containerName="payload-processing" containerID="cri-o://208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d" gracePeriod=30
Sep 18 06:50:35.517976 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.517946    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:50:35.518208 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.518182    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" podUID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" containerName="payload-pre-processing" containerID="cri-o://06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818" gracePeriod=30
Sep 18 06:50:35.620290 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.620259    2567 generic.go:356] "Generic (PLEG): container finished" podID="a4887f55-43c8-4423-a133-4c339192ea35" containerID="b25fd3c99e7d0e80e2d0d2717bc3ff9a336b2dcbbbac6cc5cfefbdc8cf1c44c4" exitCode=0
Sep 18 06:50:35.620442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.620318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" event={"ID":"a4887f55-43c8-4423-a133-4c339192ea35","Type":"ContainerDied","Data":"b25fd3c99e7d0e80e2d0d2717bc3ff9a336b2dcbbbac6cc5cfefbdc8cf1c44c4"}
Sep 18 06:50:35.799656 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.799625    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:50:35.818664 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.818550    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:50:35.826850 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.826822    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:50:35.894668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.894636    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs\") pod \"a4887f55-43c8-4423-a133-4c339192ea35\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") "
Sep 18 06:50:35.894856 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.894786    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca\") pod \"a4887f55-43c8-4423-a133-4c339192ea35\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") "
Sep 18 06:50:35.894922 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.894858    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjnh\" (UniqueName: \"kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh\") pod \"a4887f55-43c8-4423-a133-4c339192ea35\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") "
Sep 18 06:50:35.894922 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.894889    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") pod \"a4887f55-43c8-4423-a133-4c339192ea35\" (UID: \"a4887f55-43c8-4423-a133-4c339192ea35\") "
Sep 18 06:50:35.895173 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.895119    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4887f55-43c8-4423-a133-4c339192ea35" (UID: "a4887f55-43c8-4423-a133-4c339192ea35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:50:35.895297 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.895264    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4887f55-43c8-4423-a133-4c339192ea35-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:35.896924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.896900    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh" (OuterVolumeSpecName: "kube-api-access-6xjnh") pod "a4887f55-43c8-4423-a133-4c339192ea35" (UID: "a4887f55-43c8-4423-a133-4c339192ea35"). InnerVolumeSpecName "kube-api-access-6xjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:50:35.896924 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.896916    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4887f55-43c8-4423-a133-4c339192ea35" (UID: "a4887f55-43c8-4423-a133-4c339192ea35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:50:35.897044 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.896930    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4887f55-43c8-4423-a133-4c339192ea35" (UID: "a4887f55-43c8-4423-a133-4c339192ea35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:50:35.996153 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996107    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume\") pod \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") "
Sep 18 06:50:35.996332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqktr\" (UniqueName: \"kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr\") pod \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\" (UID: \"cb6a2847-e32f-4c73-bc12-3b094c4ad920\") "
Sep 18 06:50:35.996332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996240    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv2rc\" (UniqueName: \"kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc\") pod \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") "
Sep 18 06:50:35.996462 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996403    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume\") pod \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\" (UID: \"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1\") "
Sep 18 06:50:35.996521 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996490    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb6a2847-e32f-4c73-bc12-3b094c4ad920" (UID: "cb6a2847-e32f-4c73-bc12-3b094c4ad920"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:50:35.996732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996714    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb6a2847-e32f-4c73-bc12-3b094c4ad920-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:35.996803 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996735    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjnh\" (UniqueName: \"kubernetes.io/projected/a4887f55-43c8-4423-a133-4c339192ea35-kube-api-access-6xjnh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:35.996803 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996752    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:35.996803 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996763    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4887f55-43c8-4423-a133-4c339192ea35-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:35.996803 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.996727    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" (UID: "e69ea603-0964-4a1b-a3f6-adec1ee3fdd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:50:35.998179 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.998156    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr" (OuterVolumeSpecName: "kube-api-access-cqktr") pod "cb6a2847-e32f-4c73-bc12-3b094c4ad920" (UID: "cb6a2847-e32f-4c73-bc12-3b094c4ad920"). InnerVolumeSpecName "kube-api-access-cqktr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:50:35.998277 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:35.998260    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc" (OuterVolumeSpecName: "kube-api-access-mv2rc") pod "e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" (UID: "e69ea603-0964-4a1b-a3f6-adec1ee3fdd1"). InnerVolumeSpecName "kube-api-access-mv2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:50:36.097657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.097572    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:36.097657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.097601    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqktr\" (UniqueName: \"kubernetes.io/projected/cb6a2847-e32f-4c73-bc12-3b094c4ad920-kube-api-access-cqktr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:36.097657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.097613    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2rc\" (UniqueName: \"kubernetes.io/projected/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1-kube-api-access-mv2rc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:50:36.629503 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.629462    2567 generic.go:356] "Generic (PLEG): container finished" podID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" containerID="208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d" exitCode=0
Sep 18 06:50:36.629967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.629530    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"
Sep 18 06:50:36.629967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.629575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" event={"ID":"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1","Type":"ContainerDied","Data":"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"}
Sep 18 06:50:36.629967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.629658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk" event={"ID":"e69ea603-0964-4a1b-a3f6-adec1ee3fdd1","Type":"ContainerDied","Data":"35af757a7b426c6dc4b2fe116527e744becdb44ad23707f391daa3cfa8d97add"}
Sep 18 06:50:36.629967 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.629692    2567 scope.go:117] "RemoveContainer" containerID="208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"
Sep 18 06:50:36.631277 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.631210    2567 generic.go:356] "Generic (PLEG): container finished" podID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" containerID="06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818" exitCode=0
Sep 18 06:50:36.631277 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.631265    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"
Sep 18 06:50:36.631419 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.631273    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" event={"ID":"cb6a2847-e32f-4c73-bc12-3b094c4ad920","Type":"ContainerDied","Data":"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"}
Sep 18 06:50:36.631477 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.631423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg" event={"ID":"cb6a2847-e32f-4c73-bc12-3b094c4ad920","Type":"ContainerDied","Data":"ece6056d0534434c89b992307415f3cbca86803a9675eb870a54edf6924f9ebc"}
Sep 18 06:50:36.633353 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.633297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc" event={"ID":"a4887f55-43c8-4423-a133-4c339192ea35","Type":"ContainerDied","Data":"273c8ce2de3ac2d508cb0b622eb3aa625a2ee95a8b24ea624fd26bea62592af9"}
Sep 18 06:50:36.633488 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.633327    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"
Sep 18 06:50:36.641770 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.641688    2567 scope.go:117] "RemoveContainer" containerID="208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"
Sep 18 06:50:36.642031 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:50:36.642003    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d\": container with ID starting with 208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d not found: ID does not exist" containerID="208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"
Sep 18 06:50:36.642174 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.642041    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d"} err="failed to get container status \"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d\": rpc error: code = NotFound desc = could not find container \"208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d\": container with ID starting with 208b7a886de525d3859b4d4aab6483ba3dc9cb3d602d620872ebf60843bd532d not found: ID does not exist"
Sep 18 06:50:36.642174 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.642063    2567 scope.go:117] "RemoveContainer" containerID="06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"
Sep 18 06:50:36.651294 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.651272    2567 scope.go:117] "RemoveContainer" containerID="06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"
Sep 18 06:50:36.651533 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:50:36.651516    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818\": container with ID starting with 06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818 not found: ID does not exist" containerID="06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"
Sep 18 06:50:36.651585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.651538    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818"} err="failed to get container status \"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818\": rpc error: code = NotFound desc = could not find container \"06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818\": container with ID starting with 06afdf64d2ee1e3e592316b9213ce024d6bafab5f181005d3c62fc5e0972c818 not found: ID does not exist"
Sep 18 06:50:36.651585 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.651554    2567 scope.go:117] "RemoveContainer" containerID="b25fd3c99e7d0e80e2d0d2717bc3ff9a336b2dcbbbac6cc5cfefbdc8cf1c44c4"
Sep 18 06:50:36.661612 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.661591    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:50:36.667252 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.667229    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93683709-564fcc4fb9-zprdk"]
Sep 18 06:50:36.676840 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.676819    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:50:36.680896 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.680875    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93683709-68fddddbb8-rglbc"]
Sep 18 06:50:36.696978 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.696952    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:50:36.702360 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.702338    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93683709-7c56d87bc8-l5mcg"]
Sep 18 06:50:36.909660 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.909585    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4887f55-43c8-4423-a133-4c339192ea35" path="/var/lib/kubelet/pods/a4887f55-43c8-4423-a133-4c339192ea35/volumes"
Sep 18 06:50:36.909962 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.909949    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" path="/var/lib/kubelet/pods/cb6a2847-e32f-4c73-bc12-3b094c4ad920/volumes"
Sep 18 06:50:36.910286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:36.910273    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" path="/var/lib/kubelet/pods/e69ea603-0964-4a1b-a3f6-adec1ee3fdd1/volumes"
Sep 18 06:50:43.586225 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586182    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:50:43.586834 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586786    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4887f55-43c8-4423-a133-4c339192ea35" containerName="maas-api"
Sep 18 06:50:43.586834 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586808    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4887f55-43c8-4423-a133-4c339192ea35" containerName="maas-api"
Sep 18 06:50:43.586955 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586852    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" containerName="payload-pre-processing"
Sep 18 06:50:43.586955 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586861    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" containerName="payload-pre-processing"
Sep 18 06:50:43.586955 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586871    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" containerName="payload-processing"
Sep 18 06:50:43.586955 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586880    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" containerName="payload-processing"
Sep 18 06:50:43.587248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.586985    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4887f55-43c8-4423-a133-4c339192ea35" containerName="maas-api"
Sep 18 06:50:43.587248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.587001    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb6a2847-e32f-4c73-bc12-3b094c4ad920" containerName="payload-pre-processing"
Sep 18 06:50:43.587248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.587014    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69ea603-0964-4a1b-a3f6-adec1ee3fdd1" containerName="payload-processing"
Sep 18 06:50:43.591927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.591903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.594396 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.594360    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ec2077\"/\"kube-root-ca.crt\""
Sep 18 06:50:43.595534 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.595508    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ec2077\"/\"openshift-service-ca.crt\""
Sep 18 06:50:43.595658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.595607    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ec2077\"/\"default-dockercfg-b954k\""
Sep 18 06:50:43.595658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.595607    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ec2077\"/\"auth-test-model-ec2077-kserve-self-signed-certs\""
Sep 18 06:50:43.598527 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.598504    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:50:43.667645 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.667816 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667688    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.667873 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667831    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rfq\" (UniqueName: \"kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.667913 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.667959 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667925    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.668010 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.667996    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769199 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769401 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rfq\" (UniqueName: \"kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769581 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769702 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769628    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769772 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.769829 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.769765    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.771560 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.771529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.771815 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.771788    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.776841 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.776816    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rfq\" (UniqueName: \"kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq\") pod \"auth-test-model-ec2077-kserve-9b45594f9-j9wrq\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") " pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:43.903381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:43.903293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:44.172836 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:44.172782    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:50:44.175529 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:50:44.175505    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501efe12_b196_4e8c_b5a8_abcb9a96c199.slice/crio-0d1e9ea939d19c071d80f3cf7de2810f6436852aed5a940582976eeb9b18ad9f WatchSource:0}: Error finding container 0d1e9ea939d19c071d80f3cf7de2810f6436852aed5a940582976eeb9b18ad9f: Status 404 returned error can't find the container with id 0d1e9ea939d19c071d80f3cf7de2810f6436852aed5a940582976eeb9b18ad9f
Sep 18 06:50:44.680108 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:44.680061    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" event={"ID":"501efe12-b196-4e8c-b5a8-abcb9a96c199","Type":"ContainerStarted","Data":"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"}
Sep 18 06:50:44.680492 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:44.680117    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" event={"ID":"501efe12-b196-4e8c-b5a8-abcb9a96c199","Type":"ContainerStarted","Data":"0d1e9ea939d19c071d80f3cf7de2810f6436852aed5a940582976eeb9b18ad9f"}
Sep 18 06:50:44.696722 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:44.696672    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" podStartSLOduration=1.696655637 podStartE2EDuration="1.696655637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:50:44.695217323 +0000 UTC m=+1724.278602980" watchObservedRunningTime="2026-09-18 06:50:44.696655637 +0000 UTC m=+1724.280041290"
Sep 18 06:50:53.903863 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:53.903828    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:53.903863 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:53.903862    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:53.916192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:53.916165    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:50:54.743321 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:50:54.743292    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:52:00.957128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:00.957102    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:52:00.962603 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:00.962582    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:52:42.003869 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.003835    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:52:42.007750 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.007721    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.010110 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.010088    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2686de-serving-cert\""
Sep 18 06:52:42.019349 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.019325    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:52:42.123128 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.123094    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.123317 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.123175    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmz29\" (UniqueName: \"kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.123317 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.123205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.123317 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.123232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.224511 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.224473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.224701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.224550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmz29\" (UniqueName: \"kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.224701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.224583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.224701 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.224612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.225286 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.225263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.226975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.226952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.227082 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.227063    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.231964 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.231943    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmz29\" (UniqueName: \"kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29\") pod \"maas-api-e2e-api-a-2686de-759dd69b47-2hct4\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.319954 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.319866    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:42.583765 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.583683    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:52:42.587566 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.587543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.590475 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.590455    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2686de\""
Sep 18 06:52:42.590610 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.590588    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2686de-dockercfg-6dppf\""
Sep 18 06:52:42.598832 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.598769    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:52:42.601524 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.601501    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:52:42.604243 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:52:42.604214    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83db68c7_6234_4c9f_992c_b100394f9c11.slice/crio-361380cea8dc6de014f6103d2dfd457deaa0cd4c727f6a611a1c68a42b57e0ab WatchSource:0}: Error finding container 361380cea8dc6de014f6103d2dfd457deaa0cd4c727f6a611a1c68a42b57e0ab: Status 404 returned error can't find the container with id 361380cea8dc6de014f6103d2dfd457deaa0cd4c727f6a611a1c68a42b57e0ab
Sep 18 06:52:42.629076 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.629046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk6m\" (UniqueName: \"kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.629227 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.629099    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.730126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.730088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk6m\" (UniqueName: \"kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.730299 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.730189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.730876 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.730857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.737782 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.737760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk6m\" (UniqueName: \"kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m\") pod \"payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:42.899439 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:42.899341    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:43.173783 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.173757    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:52:43.176672 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:52:43.176627    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621c2f97_7305_4e57_a216_c37c6ad84b1e.slice/crio-08f4d221b97bbbd70943c6eead995c635b37f4ad63474c40532d0fa1a08ea05a WatchSource:0}: Error finding container 08f4d221b97bbbd70943c6eead995c635b37f4ad63474c40532d0fa1a08ea05a: Status 404 returned error can't find the container with id 08f4d221b97bbbd70943c6eead995c635b37f4ad63474c40532d0fa1a08ea05a
Sep 18 06:52:43.298942 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.298906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" event={"ID":"621c2f97-7305-4e57-a216-c37c6ad84b1e","Type":"ContainerStarted","Data":"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"}
Sep 18 06:52:43.299155 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.299097    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:52:43.299231 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.299171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" event={"ID":"621c2f97-7305-4e57-a216-c37c6ad84b1e","Type":"ContainerStarted","Data":"08f4d221b97bbbd70943c6eead995c635b37f4ad63474c40532d0fa1a08ea05a"}
Sep 18 06:52:43.300906 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.300878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" event={"ID":"83db68c7-6234-4c9f-992c-b100394f9c11","Type":"ContainerStarted","Data":"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"}
Sep 18 06:52:43.301036 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.300953    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" event={"ID":"83db68c7-6234-4c9f-992c-b100394f9c11","Type":"ContainerStarted","Data":"361380cea8dc6de014f6103d2dfd457deaa0cd4c727f6a611a1c68a42b57e0ab"}
Sep 18 06:52:43.301036 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.300981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:43.313496 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.313425    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" podStartSLOduration=1.313409533 podStartE2EDuration="1.313409533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:52:43.312786913 +0000 UTC m=+1842.896172569" watchObservedRunningTime="2026-09-18 06:52:43.313409533 +0000 UTC m=+1842.896795186"
Sep 18 06:52:43.328594 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:43.328548    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" podStartSLOduration=1.842279251 podStartE2EDuration="2.328533509s" totalImagesPullingTime="486.254258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:52:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:52:43.327504279 +0000 UTC m=+1842.910889938" watchObservedRunningTime="2026-09-18 06:52:43.328533509 +0000 UTC m=+1842.911919161"
Sep 18 06:52:49.312056 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:49.312025    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:52:54.308892 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:52:54.308863    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:53:16.349346 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.349310    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:53:16.353306 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.353278    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.355742 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.355722    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bcb767-serving-cert\""
Sep 18 06:53:16.362193 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.362164    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:53:16.465132 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.465098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.465326 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.465216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm26g\" (UniqueName: \"kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.465326 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.465246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.465326 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.465282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.566551 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.566518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm26g\" (UniqueName: \"kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.566551 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.566554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.566743 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:53:16.566646    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-serving-cert: secret "maas-api-e2e-api-b-bcb767-serving-cert" not found
Sep 18 06:53:16.566743 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:53:16.566701    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls podName:a72bcf7b-52ab-4a92-8383-32b1d6b2eebd nodeName:}" failed. No retries permitted until 2026-09-18 06:53:17.066684698 +0000 UTC m=+1876.650070329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls") pod "maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" (UID: "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd") : secret "maas-api-e2e-api-b-bcb767-serving-cert" not found
Sep 18 06:53:16.566870 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.566848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.567007 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.566994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.567574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.567548    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.569300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.569276    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:16.575494 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:16.575473    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm26g\" (UniqueName: \"kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:17.071998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:17.071959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:17.074634 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:17.074605    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:17.266514 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:17.266472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:17.545543 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:17.545503    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:53:17.548027 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:53:17.548000    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72bcf7b_52ab_4a92_8383_32b1d6b2eebd.slice/crio-fd31576308ea9e796b75d15821f3eb3ff7b4bc8d04d861c2dfa8b17fe6effc81 WatchSource:0}: Error finding container fd31576308ea9e796b75d15821f3eb3ff7b4bc8d04d861c2dfa8b17fe6effc81: Status 404 returned error can't find the container with id fd31576308ea9e796b75d15821f3eb3ff7b4bc8d04d861c2dfa8b17fe6effc81
Sep 18 06:53:18.486192 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:18.486153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" event={"ID":"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd","Type":"ContainerStarted","Data":"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"}
Sep 18 06:53:18.486381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:18.486219    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" event={"ID":"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd","Type":"ContainerStarted","Data":"fd31576308ea9e796b75d15821f3eb3ff7b4bc8d04d861c2dfa8b17fe6effc81"}
Sep 18 06:53:18.486381 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:18.486254    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:18.503097 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:18.503050    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" podStartSLOduration=2.062362202 podStartE2EDuration="2.503030575s" totalImagesPullingTime="440.668373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:53:18.501340255 +0000 UTC m=+1878.084725905" watchObservedRunningTime="2026-09-18 06:53:18.503030575 +0000 UTC m=+1878.086416227"
Sep 18 06:53:24.498363 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:24.498280    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:53:59.863549 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:59.863514    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:53:59.864229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:59.863705    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" podUID="83db68c7-6234-4c9f-992c-b100394f9c11" containerName="maas-api" containerID="cri-o://f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb" gracePeriod=30
Sep 18 06:53:59.908390 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:59.908356    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:53:59.908656 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:53:59.908605    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" podUID="621c2f97-7305-4e57-a216-c37c6ad84b1e" containerName="payload-pre-processing" containerID="cri-o://cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e" gracePeriod=30
Sep 18 06:54:00.156998 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.156967    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:54:00.199464 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.199440    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:54:00.209543 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.209516    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca\") pod \"83db68c7-6234-4c9f-992c-b100394f9c11\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") "
Sep 18 06:54:00.209668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.209609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls\") pod \"83db68c7-6234-4c9f-992c-b100394f9c11\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") "
Sep 18 06:54:00.209668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.209634    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmz29\" (UniqueName: \"kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29\") pod \"83db68c7-6234-4c9f-992c-b100394f9c11\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") "
Sep 18 06:54:00.209780 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.209699    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs\") pod \"83db68c7-6234-4c9f-992c-b100394f9c11\" (UID: \"83db68c7-6234-4c9f-992c-b100394f9c11\") "
Sep 18 06:54:00.209946 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.209920    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83db68c7-6234-4c9f-992c-b100394f9c11" (UID: "83db68c7-6234-4c9f-992c-b100394f9c11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:00.210051 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.210034    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83db68c7-6234-4c9f-992c-b100394f9c11-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.211911 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.211882    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "83db68c7-6234-4c9f-992c-b100394f9c11" (UID: "83db68c7-6234-4c9f-992c-b100394f9c11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:00.211911 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.211900    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83db68c7-6234-4c9f-992c-b100394f9c11" (UID: "83db68c7-6234-4c9f-992c-b100394f9c11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:00.212300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.212278    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29" (OuterVolumeSpecName: "kube-api-access-hmz29") pod "83db68c7-6234-4c9f-992c-b100394f9c11" (UID: "83db68c7-6234-4c9f-992c-b100394f9c11"). InnerVolumeSpecName "kube-api-access-hmz29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:00.310561 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310528    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume\") pod \"621c2f97-7305-4e57-a216-c37c6ad84b1e\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") "
Sep 18 06:54:00.310561 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310564    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbk6m\" (UniqueName: \"kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m\") pod \"621c2f97-7305-4e57-a216-c37c6ad84b1e\" (UID: \"621c2f97-7305-4e57-a216-c37c6ad84b1e\") "
Sep 18 06:54:00.310841 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310827    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.310841 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310841    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83db68c7-6234-4c9f-992c-b100394f9c11-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.310962 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310850    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmz29\" (UniqueName: \"kubernetes.io/projected/83db68c7-6234-4c9f-992c-b100394f9c11-kube-api-access-hmz29\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.311020 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.310963    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "621c2f97-7305-4e57-a216-c37c6ad84b1e" (UID: "621c2f97-7305-4e57-a216-c37c6ad84b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:00.312620 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.312600    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m" (OuterVolumeSpecName: "kube-api-access-sbk6m") pod "621c2f97-7305-4e57-a216-c37c6ad84b1e" (UID: "621c2f97-7305-4e57-a216-c37c6ad84b1e"). InnerVolumeSpecName "kube-api-access-sbk6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:00.412295 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.412219    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/621c2f97-7305-4e57-a216-c37c6ad84b1e-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.412295 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.412245    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk6m\" (UniqueName: \"kubernetes.io/projected/621c2f97-7305-4e57-a216-c37c6ad84b1e-kube-api-access-sbk6m\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:00.720216 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.720185    2567 generic.go:356] "Generic (PLEG): container finished" podID="621c2f97-7305-4e57-a216-c37c6ad84b1e" containerID="cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e" exitCode=0
Sep 18 06:54:00.720393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.720235    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"
Sep 18 06:54:00.720393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.720278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" event={"ID":"621c2f97-7305-4e57-a216-c37c6ad84b1e","Type":"ContainerDied","Data":"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"}
Sep 18 06:54:00.720393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.720327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm" event={"ID":"621c2f97-7305-4e57-a216-c37c6ad84b1e","Type":"ContainerDied","Data":"08f4d221b97bbbd70943c6eead995c635b37f4ad63474c40532d0fa1a08ea05a"}
Sep 18 06:54:00.720393 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.720351    2567 scope.go:117] "RemoveContainer" containerID="cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"
Sep 18 06:54:00.721619 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.721596    2567 generic.go:356] "Generic (PLEG): container finished" podID="83db68c7-6234-4c9f-992c-b100394f9c11" containerID="f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb" exitCode=0
Sep 18 06:54:00.721733 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.721622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" event={"ID":"83db68c7-6234-4c9f-992c-b100394f9c11","Type":"ContainerDied","Data":"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"}
Sep 18 06:54:00.721733 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.721653    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"
Sep 18 06:54:00.721733 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.721670    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4" event={"ID":"83db68c7-6234-4c9f-992c-b100394f9c11","Type":"ContainerDied","Data":"361380cea8dc6de014f6103d2dfd457deaa0cd4c727f6a611a1c68a42b57e0ab"}
Sep 18 06:54:00.729250 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.729193    2567 scope.go:117] "RemoveContainer" containerID="cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"
Sep 18 06:54:00.729557 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:00.729535    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e\": container with ID starting with cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e not found: ID does not exist" containerID="cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"
Sep 18 06:54:00.729659 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.729566    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e"} err="failed to get container status \"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e\": rpc error: code = NotFound desc = could not find container \"cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e\": container with ID starting with cd51428fb2e5ff6329af1fcb6acfc6c9de2c5a38abb9cdec32697e028af6a70e not found: ID does not exist"
Sep 18 06:54:00.729659 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.729600    2567 scope.go:117] "RemoveContainer" containerID="f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"
Sep 18 06:54:00.737923 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.737905    2567 scope.go:117] "RemoveContainer" containerID="f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"
Sep 18 06:54:00.738220 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:00.738198    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb\": container with ID starting with f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb not found: ID does not exist" containerID="f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"
Sep 18 06:54:00.738290 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.738226    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb"} err="failed to get container status \"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb\": rpc error: code = NotFound desc = could not find container \"f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb\": container with ID starting with f25c8acb928a99ef7fa8595f9b71b6e94b22388dc96756725e6f649d2636dcdb not found: ID does not exist"
Sep 18 06:54:00.746437 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.746415    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:54:00.749537 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.749516    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2686de-759dd69b47-2hct4"]
Sep 18 06:54:00.759574 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.759555    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:54:00.768516 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.768496    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2686de-667c569ffd-brnhm"]
Sep 18 06:54:00.911330 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.911299    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="621c2f97-7305-4e57-a216-c37c6ad84b1e" path="/var/lib/kubelet/pods/621c2f97-7305-4e57-a216-c37c6ad84b1e/volumes"
Sep 18 06:54:00.911727 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:00.911712    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83db68c7-6234-4c9f-992c-b100394f9c11" path="/var/lib/kubelet/pods/83db68c7-6234-4c9f-992c-b100394f9c11/volumes"
Sep 18 06:54:14.793501 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:14.793415    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:54:14.816074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:14.816043    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:54:23.067056 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.067024    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:54:23.067868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.067244    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" podUID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" containerName="maas-api" containerID="cri-o://a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809" gracePeriod=30
Sep 18 06:54:23.322461 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.322400    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:54:23.431891 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.431851    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs\") pod \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") "
Sep 18 06:54:23.432086 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.431952    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") pod \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") "
Sep 18 06:54:23.432086 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.431987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca\") pod \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") "
Sep 18 06:54:23.432086 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.432039    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm26g\" (UniqueName: \"kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g\") pod \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\" (UID: \"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd\") "
Sep 18 06:54:23.432421 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.432382    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" (UID: "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:23.434211 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.434154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g" (OuterVolumeSpecName: "kube-api-access-vm26g") pod "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" (UID: "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd"). InnerVolumeSpecName "kube-api-access-vm26g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:23.434211 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.434172    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" (UID: "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:23.434341 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.434225    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" (UID: "a72bcf7b-52ab-4a92-8383-32b1d6b2eebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:23.533625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.533589    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm26g\" (UniqueName: \"kubernetes.io/projected/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-kube-api-access-vm26g\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:23.533625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.533619    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:23.533625 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.533628    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:23.533843 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.533636    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:23.840652 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.840618    2567 generic.go:356] "Generic (PLEG): container finished" podID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" containerID="a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809" exitCode=0
Sep 18 06:54:23.840820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.840664    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"
Sep 18 06:54:23.840820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.840703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" event={"ID":"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd","Type":"ContainerDied","Data":"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"}
Sep 18 06:54:23.840820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.840755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb" event={"ID":"a72bcf7b-52ab-4a92-8383-32b1d6b2eebd","Type":"ContainerDied","Data":"fd31576308ea9e796b75d15821f3eb3ff7b4bc8d04d861c2dfa8b17fe6effc81"}
Sep 18 06:54:23.840820 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.840780    2567 scope.go:117] "RemoveContainer" containerID="a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"
Sep 18 06:54:23.850310 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.850294    2567 scope.go:117] "RemoveContainer" containerID="a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"
Sep 18 06:54:23.850559 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:23.850539    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809\": container with ID starting with a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809 not found: ID does not exist" containerID="a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"
Sep 18 06:54:23.850610 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.850568    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809"} err="failed to get container status \"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809\": rpc error: code = NotFound desc = could not find container \"a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809\": container with ID starting with a3419e21517c2895de4c6058464160b6d2f3f4d00f4b6e1dfa3f972ce7081809 not found: ID does not exist"
Sep 18 06:54:23.861244 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.861225    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:54:23.864086 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:23.864065    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb767-56bbfcc7b4-68xpb"]
Sep 18 06:54:24.910273 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:24.910240    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" path="/var/lib/kubelet/pods/a72bcf7b-52ab-4a92-8383-32b1d6b2eebd/volumes"
Sep 18 06:54:29.812715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:29.812655    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" podUID="fdd08d85-617a-4c11-861a-16d92183cf4a" containerName="main" containerID="cri-o://5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c" gracePeriod=45
Sep 18 06:54:29.835392 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:29.835339    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" podUID="972da67d-dbc6-4268-a99f-1aa010bf796f" containerName="main" containerID="cri-o://e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6" gracePeriod=45
Sep 18 06:54:30.102609 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.102579    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:54:30.136375 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.136342    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:54:30.196529 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196486    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.196671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196546    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.196671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196608    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z2jw\" (UniqueName: \"kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.196671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196635    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.196671 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196661    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196706    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196759    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196775    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196798    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196821    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25vgt\" (UniqueName: \"kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.196872 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196866    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm\") pod \"fdd08d85-617a-4c11-861a-16d92183cf4a\" (UID: \"fdd08d85-617a-4c11-861a-16d92183cf4a\") "
Sep 18 06:54:30.197210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196896    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home\") pod \"972da67d-dbc6-4268-a99f-1aa010bf796f\" (UID: \"972da67d-dbc6-4268-a99f-1aa010bf796f\") "
Sep 18 06:54:30.197210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.196984    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.197334 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197265    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home" (OuterVolumeSpecName: "home") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.197427 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197391    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home" (OuterVolumeSpecName: "home") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.197608 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197584    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.197793 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197662    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.197793 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197683    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.197793 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197695    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.197793 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197758    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache" (OuterVolumeSpecName: "model-cache") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.198198 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.197792    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.199949 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.199918    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw" (OuterVolumeSpecName: "kube-api-access-4z2jw") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "kube-api-access-4z2jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:30.200312 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.200288    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm" (OuterVolumeSpecName: "dshm") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.200423 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.200360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt" (OuterVolumeSpecName: "kube-api-access-25vgt") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "kube-api-access-25vgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:30.200423 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.200400    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdd08d85-617a-4c11-861a-16d92183cf4a" (UID: "fdd08d85-617a-4c11-861a-16d92183cf4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:30.200615 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.200587    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:30.200615 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.200606    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm" (OuterVolumeSpecName: "dshm") pod "972da67d-dbc6-4268-a99f-1aa010bf796f" (UID: "972da67d-dbc6-4268-a99f-1aa010bf796f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:30.298704 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298666    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2jw\" (UniqueName: \"kubernetes.io/projected/fdd08d85-617a-4c11-861a-16d92183cf4a-kube-api-access-4z2jw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.298704 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298697    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/972da67d-dbc6-4268-a99f-1aa010bf796f-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.298704 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298708    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdd08d85-617a-4c11-861a-16d92183cf4a-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.298704 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298716    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.299046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298726    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.299046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298734    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25vgt\" (UniqueName: \"kubernetes.io/projected/972da67d-dbc6-4268-a99f-1aa010bf796f-kube-api-access-25vgt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.299046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298742    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdd08d85-617a-4c11-861a-16d92183cf4a-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.299046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298751    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.299046 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.298758    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/972da67d-dbc6-4268-a99f-1aa010bf796f-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:30.878790 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.878758    2567 generic.go:356] "Generic (PLEG): container finished" podID="fdd08d85-617a-4c11-861a-16d92183cf4a" containerID="5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c" exitCode=0
Sep 18 06:54:30.879187 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.878823    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"
Sep 18 06:54:30.879187 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.878850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" event={"ID":"fdd08d85-617a-4c11-861a-16d92183cf4a","Type":"ContainerDied","Data":"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"}
Sep 18 06:54:30.879187 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.878892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47" event={"ID":"fdd08d85-617a-4c11-861a-16d92183cf4a","Type":"ContainerDied","Data":"deaf6bcda0f71bdb36a13a477649f1e18c75abbc1ece2b693cf8a7fafe5a6f28"}
Sep 18 06:54:30.879187 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.878913    2567 scope.go:117] "RemoveContainer" containerID="5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"
Sep 18 06:54:30.880730 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.880707    2567 generic.go:356] "Generic (PLEG): container finished" podID="972da67d-dbc6-4268-a99f-1aa010bf796f" containerID="e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6" exitCode=0
Sep 18 06:54:30.880868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.880735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" event={"ID":"972da67d-dbc6-4268-a99f-1aa010bf796f","Type":"ContainerDied","Data":"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"}
Sep 18 06:54:30.880868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.880761    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"
Sep 18 06:54:30.880868 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.880768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck" event={"ID":"972da67d-dbc6-4268-a99f-1aa010bf796f","Type":"ContainerDied","Data":"bf093d5eebfdac0dffda669949a435e1051e3510358b703e95c8584df3a86d0a"}
Sep 18 06:54:30.888753 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.888687    2567 scope.go:117] "RemoveContainer" containerID="5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"
Sep 18 06:54:30.889073 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:30.889047    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c\": container with ID starting with 5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c not found: ID does not exist" containerID="5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"
Sep 18 06:54:30.889191 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.889085    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c"} err="failed to get container status \"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c\": rpc error: code = NotFound desc = could not find container \"5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c\": container with ID starting with 5bf5c610c9092ede88de54e6e5b404ccf1117772b19581461d84a5c08aa5027c not found: ID does not exist"
Sep 18 06:54:30.889191 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.889107    2567 scope.go:117] "RemoveContainer" containerID="e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"
Sep 18 06:54:30.897459 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.897444    2567 scope.go:117] "RemoveContainer" containerID="e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"
Sep 18 06:54:30.897717 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:30.897698    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6\": container with ID starting with e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6 not found: ID does not exist" containerID="e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"
Sep 18 06:54:30.897813 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.897720    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6"} err="failed to get container status \"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6\": rpc error: code = NotFound desc = could not find container \"e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6\": container with ID starting with e444ea98b32d5f5d09d3ce8eaf377db393b96575f76a977db866c0cd873225b6 not found: ID does not exist"
Sep 18 06:54:30.901638 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.901614    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:54:30.909657 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.909637    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-946609/distinct-w0-946609-kserve-7d7fbf4c49-kqv47"]
Sep 18 06:54:30.919570 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.919549    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:54:30.923002 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:30.922977    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-946609/premium-w0-946609-kserve-5478549b46-blnck"]
Sep 18 06:54:32.910491 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:32.910460    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972da67d-dbc6-4268-a99f-1aa010bf796f" path="/var/lib/kubelet/pods/972da67d-dbc6-4268-a99f-1aa010bf796f/volumes"
Sep 18 06:54:32.910898 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:32.910883    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd08d85-617a-4c11-861a-16d92183cf4a" path="/var/lib/kubelet/pods/fdd08d85-617a-4c11-861a-16d92183cf4a/volumes"
Sep 18 06:54:37.767569 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:37.767530    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:54:49.928350 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:49.928314    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:54:49.928841 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:49.928577    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" podUID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" containerName="maas-api" containerID="cri-o://b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3" gracePeriod=30
Sep 18 06:54:49.983336 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:49.983286    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:54:49.983691 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:49.983531    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" podUID="d833dc52-82f6-465e-b59a-412709b25164" containerName="payload-pre-processing" containerID="cri-o://69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052" gracePeriod=30
Sep 18 06:54:50.238431 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.238405    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:54:50.267860 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.267829    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:54:50.290226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.289920    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mql9t\" (UniqueName: \"kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t\") pod \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") "
Sep 18 06:54:50.290226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.289988    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs\") pod \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") "
Sep 18 06:54:50.290226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.290036    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca\") pod \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") "
Sep 18 06:54:50.290226 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.290065    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls\") pod \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\" (UID: \"863c0a4a-05ce-46dd-b947-c01c9641e9cb\") "
Sep 18 06:54:50.290865 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.290822    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "863c0a4a-05ce-46dd-b947-c01c9641e9cb" (UID: "863c0a4a-05ce-46dd-b947-c01c9641e9cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:50.293055 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.293027    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t" (OuterVolumeSpecName: "kube-api-access-mql9t") pod "863c0a4a-05ce-46dd-b947-c01c9641e9cb" (UID: "863c0a4a-05ce-46dd-b947-c01c9641e9cb"). InnerVolumeSpecName "kube-api-access-mql9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:50.293339 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.293306    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "863c0a4a-05ce-46dd-b947-c01c9641e9cb" (UID: "863c0a4a-05ce-46dd-b947-c01c9641e9cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:50.293771 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.293745    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "863c0a4a-05ce-46dd-b947-c01c9641e9cb" (UID: "863c0a4a-05ce-46dd-b947-c01c9641e9cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:50.390961 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.390925    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume\") pod \"d833dc52-82f6-465e-b59a-412709b25164\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") "
Sep 18 06:54:50.391124 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.390977    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd56z\" (UniqueName: \"kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z\") pod \"d833dc52-82f6-465e-b59a-412709b25164\" (UID: \"d833dc52-82f6-465e-b59a-412709b25164\") "
Sep 18 06:54:50.391396 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.391363    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d833dc52-82f6-465e-b59a-412709b25164" (UID: "d833dc52-82f6-465e-b59a-412709b25164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:50.391547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.391415    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.391547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.391426    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863c0a4a-05ce-46dd-b947-c01c9641e9cb-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.391547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.391436    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863c0a4a-05ce-46dd-b947-c01c9641e9cb-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.391547 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.391445    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mql9t\" (UniqueName: \"kubernetes.io/projected/863c0a4a-05ce-46dd-b947-c01c9641e9cb-kube-api-access-mql9t\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.393098 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.393070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z" (OuterVolumeSpecName: "kube-api-access-jd56z") pod "d833dc52-82f6-465e-b59a-412709b25164" (UID: "d833dc52-82f6-465e-b59a-412709b25164"). InnerVolumeSpecName "kube-api-access-jd56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:50.492792 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.492753    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d833dc52-82f6-465e-b59a-412709b25164-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.492792 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.492783    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd56z\" (UniqueName: \"kubernetes.io/projected/d833dc52-82f6-465e-b59a-412709b25164-kube-api-access-jd56z\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:50.996530 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.996494    2567 generic.go:356] "Generic (PLEG): container finished" podID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" containerID="b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3" exitCode=0
Sep 18 06:54:50.996975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.996540    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"
Sep 18 06:54:50.996975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.996534    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" event={"ID":"863c0a4a-05ce-46dd-b947-c01c9641e9cb","Type":"ContainerDied","Data":"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"}
Sep 18 06:54:50.996975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.996682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7" event={"ID":"863c0a4a-05ce-46dd-b947-c01c9641e9cb","Type":"ContainerDied","Data":"e49f1bfe6be3b31e7556b89cbdf7ff012391e6bc7b9cae53c041ca89b2013841"}
Sep 18 06:54:50.996975 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.996713    2567 scope.go:117] "RemoveContainer" containerID="b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"
Sep 18 06:54:50.998351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.998194    2567 generic.go:356] "Generic (PLEG): container finished" podID="d833dc52-82f6-465e-b59a-412709b25164" containerID="69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052" exitCode=0
Sep 18 06:54:50.998351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.998223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" event={"ID":"d833dc52-82f6-465e-b59a-412709b25164","Type":"ContainerDied","Data":"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"}
Sep 18 06:54:50.998351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.998256    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"
Sep 18 06:54:50.998351 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:50.998264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg" event={"ID":"d833dc52-82f6-465e-b59a-412709b25164","Type":"ContainerDied","Data":"559ae90f75476abab9149ef6bb0a0620c0c9486eebdf0472d1172a30bc60086e"}
Sep 18 06:54:51.008482 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.008462    2567 scope.go:117] "RemoveContainer" containerID="b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"
Sep 18 06:54:51.008827 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:51.008802    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3\": container with ID starting with b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3 not found: ID does not exist" containerID="b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"
Sep 18 06:54:51.008927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.008839    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3"} err="failed to get container status \"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3\": rpc error: code = NotFound desc = could not find container \"b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3\": container with ID starting with b408d7c1a7c529ad3b9905ad658cf5313ba60847633d2212e8bd9b504ac02cf3 not found: ID does not exist"
Sep 18 06:54:51.008927 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.008866    2567 scope.go:117] "RemoveContainer" containerID="69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"
Sep 18 06:54:51.023332 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.023290    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:54:51.023786 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.023752    2567 scope.go:117] "RemoveContainer" containerID="69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"
Sep 18 06:54:51.024183 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:54:51.024121    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052\": container with ID starting with 69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052 not found: ID does not exist" containerID="69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"
Sep 18 06:54:51.024274 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.024195    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052"} err="failed to get container status \"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052\": rpc error: code = NotFound desc = could not find container \"69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052\": container with ID starting with 69fb3bd9df9a6686424b3037499b7a4549993ce9ab80db727241ac20e98cb052 not found: ID does not exist"
Sep 18 06:54:51.029052 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.029000    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-654463-599bb7b746-lbfl7"]
Sep 18 06:54:51.051710 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.051679    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:54:51.080070 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:51.080035    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-654463-64f88b955d-dctsg"]
Sep 18 06:54:52.786822 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:52.786753    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" podUID="19cce629-d9ba-4379-9daf-c762bbce6c73" containerName="main" containerID="cri-o://7eee84d7042415eaec3cb541a2a3f15cf564a3808f498a14525c952880649680" gracePeriod=45
Sep 18 06:54:52.911080 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:52.911038    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" path="/var/lib/kubelet/pods/863c0a4a-05ce-46dd-b947-c01c9641e9cb/volumes"
Sep 18 06:54:52.911637 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:52.911616    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d833dc52-82f6-465e-b59a-412709b25164" path="/var/lib/kubelet/pods/d833dc52-82f6-465e-b59a-412709b25164/volumes"
Sep 18 06:54:53.013453 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.013423    2567 generic.go:356] "Generic (PLEG): container finished" podID="19cce629-d9ba-4379-9daf-c762bbce6c73" containerID="7eee84d7042415eaec3cb541a2a3f15cf564a3808f498a14525c952880649680" exitCode=0
Sep 18 06:54:53.013570 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.013490    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" event={"ID":"19cce629-d9ba-4379-9daf-c762bbce6c73","Type":"ContainerDied","Data":"7eee84d7042415eaec3cb541a2a3f15cf564a3808f498a14525c952880649680"}
Sep 18 06:54:53.035299 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.035274    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:54:53.115146 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115051    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115285 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115172    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrrbv\" (UniqueName: \"kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115285 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115208    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115285 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115261    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115422 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115295    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115422 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home\") pod \"19cce629-d9ba-4379-9daf-c762bbce6c73\" (UID: \"19cce629-d9ba-4379-9daf-c762bbce6c73\") "
Sep 18 06:54:53.115552 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115520    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache" (OuterVolumeSpecName: "model-cache") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:53.115699 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115669    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home" (OuterVolumeSpecName: "home") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:53.115699 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115682    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:53.115953 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.115756    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:53.117237 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.117203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:53.117372 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.117354    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm" (OuterVolumeSpecName: "dshm") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:54:53.117422 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.117387    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv" (OuterVolumeSpecName: "kube-api-access-hrrbv") pod "19cce629-d9ba-4379-9daf-c762bbce6c73" (UID: "19cce629-d9ba-4379-9daf-c762bbce6c73"). InnerVolumeSpecName "kube-api-access-hrrbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:53.216248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.216217    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrbv\" (UniqueName: \"kubernetes.io/projected/19cce629-d9ba-4379-9daf-c762bbce6c73-kube-api-access-hrrbv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:53.216248 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.216247    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:53.216447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.216283    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:53.216447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.216295    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19cce629-d9ba-4379-9daf-c762bbce6c73-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:53.216447 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:53.216308    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19cce629-d9ba-4379-9daf-c762bbce6c73-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:54.020467 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.020435    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz" event={"ID":"19cce629-d9ba-4379-9daf-c762bbce6c73","Type":"ContainerDied","Data":"873060bbc8ff1254aa59fd5a892eb15b2d2fd3acd515a1626acce1a7ec786c11"}
Sep 18 06:54:54.020467 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.020458    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"
Sep 18 06:54:54.020928 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.020483    2567 scope.go:117] "RemoveContainer" containerID="7eee84d7042415eaec3cb541a2a3f15cf564a3808f498a14525c952880649680"
Sep 18 06:54:54.044215 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.044191    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:54:54.047732 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.047712    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-654463/distinct-w4-654463-kserve-5f44464c77-t6hnz"]
Sep 18 06:54:54.910828 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:54.910790    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19cce629-d9ba-4379-9daf-c762bbce6c73" path="/var/lib/kubelet/pods/19cce629-d9ba-4379-9daf-c762bbce6c73/volumes"
Sep 18 06:54:59.205605 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.205566    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:54:59.206066 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.205832    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" podUID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" containerName="maas-api" containerID="cri-o://a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c" gracePeriod=30
Sep 18 06:54:59.463914 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.463856    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:54:59.591952 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.591919    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") pod \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") "
Sep 18 06:54:59.592126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.592016    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca\") pod \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") "
Sep 18 06:54:59.592126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.592053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs\") pod \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") "
Sep 18 06:54:59.592126 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.592094    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgrvp\" (UniqueName: \"kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp\") pod \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\" (UID: \"633cece7-7c06-4ef4-84eb-9a68ff8cf796\") "
Sep 18 06:54:59.592476 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.592437    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "633cece7-7c06-4ef4-84eb-9a68ff8cf796" (UID: "633cece7-7c06-4ef4-84eb-9a68ff8cf796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 06:54:59.594303 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.594276    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "633cece7-7c06-4ef4-84eb-9a68ff8cf796" (UID: "633cece7-7c06-4ef4-84eb-9a68ff8cf796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:59.594303 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.594295    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "633cece7-7c06-4ef4-84eb-9a68ff8cf796" (UID: "633cece7-7c06-4ef4-84eb-9a68ff8cf796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:54:59.594494 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.594345    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp" (OuterVolumeSpecName: "kube-api-access-qgrvp") pod "633cece7-7c06-4ef4-84eb-9a68ff8cf796" (UID: "633cece7-7c06-4ef4-84eb-9a68ff8cf796"). InnerVolumeSpecName "kube-api-access-qgrvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:54:59.693591 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.693557    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:59.693591 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.693584    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633cece7-7c06-4ef4-84eb-9a68ff8cf796-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:59.693591 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.693596    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633cece7-7c06-4ef4-84eb-9a68ff8cf796-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:54:59.693821 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:54:59.693604    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrvp\" (UniqueName: \"kubernetes.io/projected/633cece7-7c06-4ef4-84eb-9a68ff8cf796-kube-api-access-qgrvp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:00.051846 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.051810    2567 generic.go:356] "Generic (PLEG): container finished" podID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" containerID="a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c" exitCode=0
Sep 18 06:55:00.052074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.051862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" event={"ID":"633cece7-7c06-4ef4-84eb-9a68ff8cf796","Type":"ContainerDied","Data":"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"}
Sep 18 06:55:00.052074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.051864    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"
Sep 18 06:55:00.052074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.051904    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt" event={"ID":"633cece7-7c06-4ef4-84eb-9a68ff8cf796","Type":"ContainerDied","Data":"de78cfa5a2674ac839b8a18035714d607d4fbf55df3c10b28ac91d51b9ddc49b"}
Sep 18 06:55:00.052074 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.051926    2567 scope.go:117] "RemoveContainer" containerID="a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"
Sep 18 06:55:00.061500 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.061483    2567 scope.go:117] "RemoveContainer" containerID="a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"
Sep 18 06:55:00.061766 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:55:00.061747    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c\": container with ID starting with a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c not found: ID does not exist" containerID="a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"
Sep 18 06:55:00.061826 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.061779    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c"} err="failed to get container status \"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c\": rpc error: code = NotFound desc = could not find container \"a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c\": container with ID starting with a76819d68dbf8594924c812a19be9b7c9410bc63593f9530c188797c720be43c not found: ID does not exist"
Sep 18 06:55:00.075757 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.075733    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:55:00.078380 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.078359    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-87d618-7fdf64f576-zlfpt"]
Sep 18 06:55:00.913983 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:00.913950    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" path="/var/lib/kubelet/pods/633cece7-7c06-4ef4-84eb-9a68ff8cf796/volumes"
Sep 18 06:55:31.396329 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:31.396295    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:55:46.415890 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.415828    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" podUID="501efe12-b196-4e8c-b5a8-abcb9a96c199" containerName="main" containerID="cri-o://83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5" gracePeriod=45
Sep 18 06:55:46.666854 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.666796    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:55:46.836465 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836431    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.836658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6rfq\" (UniqueName: \"kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.836658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836546    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.836658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836593    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.836658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836617    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.836658 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.836648    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache\") pod \"501efe12-b196-4e8c-b5a8-abcb9a96c199\" (UID: \"501efe12-b196-4e8c-b5a8-abcb9a96c199\") "
Sep 18 06:55:46.837159 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.837117    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache" (OuterVolumeSpecName: "model-cache") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:55:46.840292 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.840262    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home" (OuterVolumeSpecName: "home") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:55:46.840388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.840267    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:55:46.840388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.840348    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 06:55:46.840388 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.840375    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm" (OuterVolumeSpecName: "dshm") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 06:55:46.841668 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.841645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq" (OuterVolumeSpecName: "kube-api-access-s6rfq") pod "501efe12-b196-4e8c-b5a8-abcb9a96c199" (UID: "501efe12-b196-4e8c-b5a8-abcb9a96c199"). InnerVolumeSpecName "kube-api-access-s6rfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 06:55:46.937578 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937551    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501efe12-b196-4e8c-b5a8-abcb9a96c199-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:46.937578 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937576    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rfq\" (UniqueName: \"kubernetes.io/projected/501efe12-b196-4e8c-b5a8-abcb9a96c199-kube-api-access-s6rfq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:46.937745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937586    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:46.937745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937595    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:46.937745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937602    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:46.937745 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:46.937609    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501efe12-b196-4e8c-b5a8-abcb9a96c199-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 06:55:47.281218 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.281179    2567 generic.go:356] "Generic (PLEG): container finished" podID="501efe12-b196-4e8c-b5a8-abcb9a96c199" containerID="83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5" exitCode=0
Sep 18 06:55:47.281424 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.281239    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"
Sep 18 06:55:47.281424 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.281272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" event={"ID":"501efe12-b196-4e8c-b5a8-abcb9a96c199","Type":"ContainerDied","Data":"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"}
Sep 18 06:55:47.281424 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.281327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq" event={"ID":"501efe12-b196-4e8c-b5a8-abcb9a96c199","Type":"ContainerDied","Data":"0d1e9ea939d19c071d80f3cf7de2810f6436852aed5a940582976eeb9b18ad9f"}
Sep 18 06:55:47.281424 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.281349    2567 scope.go:117] "RemoveContainer" containerID="83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"
Sep 18 06:55:47.289826 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.289810    2567 scope.go:117] "RemoveContainer" containerID="83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"
Sep 18 06:55:47.290095 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:55:47.290067    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5\": container with ID starting with 83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5 not found: ID does not exist" containerID="83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"
Sep 18 06:55:47.290210 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.290105    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5"} err="failed to get container status \"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5\": rpc error: code = NotFound desc = could not find container \"83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5\": container with ID starting with 83634cdaad0b1fc105b6ef54a4ec508a1749db0ae996f6165cfe4ec7c7291de5 not found: ID does not exist"
Sep 18 06:55:47.298586 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.298551    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:55:47.300357 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:47.300338    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ec2077/auth-test-model-ec2077-kserve-9b45594f9-j9wrq"]
Sep 18 06:55:48.909572 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:55:48.909539    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501efe12-b196-4e8c-b5a8-abcb9a96c199" path="/var/lib/kubelet/pods/501efe12-b196-4e8c-b5a8-abcb9a96c199/volumes"
Sep 18 06:57:00.986715 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:57:00.986687    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:57:00.992734 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:57:00.992710    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 06:58:45.914009 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.913972    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914379    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd08d85-617a-4c11-861a-16d92183cf4a" containerName="main"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914393    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd08d85-617a-4c11-861a-16d92183cf4a" containerName="main"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914403    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d833dc52-82f6-465e-b59a-412709b25164" containerName="payload-pre-processing"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914409    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d833dc52-82f6-465e-b59a-412709b25164" containerName="payload-pre-processing"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914419    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" containerName="maas-api"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914426    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" containerName="maas-api"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914435    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501efe12-b196-4e8c-b5a8-abcb9a96c199" containerName="main"
Sep 18 06:58:45.914442 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914440    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="501efe12-b196-4e8c-b5a8-abcb9a96c199" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914450    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19cce629-d9ba-4379-9daf-c762bbce6c73" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914456    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19cce629-d9ba-4379-9daf-c762bbce6c73" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914461    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621c2f97-7305-4e57-a216-c37c6ad84b1e" containerName="payload-pre-processing"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914466    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="621c2f97-7305-4e57-a216-c37c6ad84b1e" containerName="payload-pre-processing"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914474    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83db68c7-6234-4c9f-992c-b100394f9c11" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914479    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="83db68c7-6234-4c9f-992c-b100394f9c11" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914487    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914492    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914498    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914503    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914513    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972da67d-dbc6-4268-a99f-1aa010bf796f" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914518    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="972da67d-dbc6-4268-a99f-1aa010bf796f" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914571    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a72bcf7b-52ab-4a92-8383-32b1d6b2eebd" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914581    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd08d85-617a-4c11-861a-16d92183cf4a" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914587    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="633cece7-7c06-4ef4-84eb-9a68ff8cf796" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914592    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="83db68c7-6234-4c9f-992c-b100394f9c11" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914598    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="19cce629-d9ba-4379-9daf-c762bbce6c73" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914603    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="863c0a4a-05ce-46dd-b947-c01c9641e9cb" containerName="maas-api"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914607    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="501efe12-b196-4e8c-b5a8-abcb9a96c199" containerName="main"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914612    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="621c2f97-7305-4e57-a216-c37c6ad84b1e" containerName="payload-pre-processing"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914618    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d833dc52-82f6-465e-b59a-412709b25164" containerName="payload-pre-processing"
Sep 18 06:58:45.914751 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.914624    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="972da67d-dbc6-4268-a99f-1aa010bf796f" containerName="main"
Sep 18 06:58:45.917706 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.917689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:45.920300 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.920222    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a7bd58-serving-cert\""
Sep 18 06:58:45.929542 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:45.929517    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 06:58:46.053033 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.052999    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.053234 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.053042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tl75\" (UniqueName: \"kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.053234 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.053092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.053234 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.053214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.154229 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.154196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.154410 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.154249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.154410 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.154284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.154410 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.154306    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tl75\" (UniqueName: \"kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.154410 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:58:46.154339    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-serving-cert: secret "maas-api-e2e-ipp-a-a7bd58-serving-cert" not found
Sep 18 06:58:46.154410 ip-10-0-143-147 kubenswrapper[2567]: E0918 06:58:46.154409    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls podName:3e0afe96-74f3-4135-8e83-12c8ef2fc13d nodeName:}" failed. No retries permitted until 2026-09-18 06:58:46.6543938 +0000 UTC m=+2206.237779435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls") pod "maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" (UID: "3e0afe96-74f3-4135-8e83-12c8ef2fc13d") : secret "maas-api-e2e-ipp-a-a7bd58-serving-cert" not found
Sep 18 06:58:46.154931 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.154910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.156769 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.156746    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.162382 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.162358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tl75\" (UniqueName: \"kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.658707 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.658665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.661157 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.661107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:46.829503 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:46.829472    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:47.101788 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:47.101753    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 06:58:47.103822 ip-10-0-143-147 kubenswrapper[2567]: W0918 06:58:47.103782    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0afe96_74f3_4135_8e83_12c8ef2fc13d.slice/crio-1eb205e8a225d3e70ad9714e2b28e5a2d34f6ad932666be49a7bf8d00742e5e4 WatchSource:0}: Error finding container 1eb205e8a225d3e70ad9714e2b28e5a2d34f6ad932666be49a7bf8d00742e5e4: Status 404 returned error can't find the container with id 1eb205e8a225d3e70ad9714e2b28e5a2d34f6ad932666be49a7bf8d00742e5e4
Sep 18 06:58:47.105269 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:47.105253    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 06:58:47.148670 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:47.148640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" event={"ID":"3e0afe96-74f3-4135-8e83-12c8ef2fc13d","Type":"ContainerStarted","Data":"1eb205e8a225d3e70ad9714e2b28e5a2d34f6ad932666be49a7bf8d00742e5e4"}
Sep 18 06:58:48.155281 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:48.155229    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" event={"ID":"3e0afe96-74f3-4135-8e83-12c8ef2fc13d","Type":"ContainerStarted","Data":"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"}
Sep 18 06:58:48.155651 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:48.155424    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 06:58:48.172963 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:48.172917    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" podStartSLOduration=2.675850197 podStartE2EDuration="3.172898419s" totalImagesPullingTime="497.048222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 06:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 06:58:48.170352807 +0000 UTC m=+2207.753738460" watchObservedRunningTime="2026-09-18 06:58:48.172898419 +0000 UTC m=+2207.756284076"
Sep 18 06:58:54.166301 ip-10-0-143-147 kubenswrapper[2567]: I0918 06:58:54.166270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 07:00:26.129890 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.129852    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:00:26.133605 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.133580    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.136216 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.136194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-75ce90-serving-cert\""
Sep 18 07:00:26.143675 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.143650    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:00:26.231672 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.231638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.231672 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.231674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.231894 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.231705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nql7d\" (UniqueName: \"kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.231894 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.231778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.332568 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.332531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.332568 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.332567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.332824 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.332609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nql7d\" (UniqueName: \"kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.332824 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.332738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.333323 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.333304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.335022 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.335002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.335115 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.335100    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.340375 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.340345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nql7d\" (UniqueName: \"kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d\") pod \"maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.446524 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.446498    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:26.718770 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.718743    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:00:26.721487 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:00:26.721453    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84176077_bde3_4204_8f7a_c54e40574744.slice/crio-82efc394ab1fbc1afd542bc7f50b597cb1bfd210330513fb2e4764876cceb6b3 WatchSource:0}: Error finding container 82efc394ab1fbc1afd542bc7f50b597cb1bfd210330513fb2e4764876cceb6b3: Status 404 returned error can't find the container with id 82efc394ab1fbc1afd542bc7f50b597cb1bfd210330513fb2e4764876cceb6b3
Sep 18 07:00:26.722868 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.722838    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:00:26.728098 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.728077    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.730477 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.730455    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-75ce90-dockercfg-fsnpx\""
Sep 18 07:00:26.730562 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.730477    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-75ce90\""
Sep 18 07:00:26.737506 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.737469    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:00:26.837323 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.837288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgt5\" (UniqueName: \"kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.837514 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.837413    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.938431 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.938397    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgt5\" (UniqueName: \"kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.938598 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.938469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.939103 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.939082    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:26.946056 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:26.946035    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgt5\" (UniqueName: \"kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5\") pod \"payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:27.042306 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.042211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:27.312817 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.312792    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:00:27.314529 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:00:27.314498    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566c6592_1f21_4e44_9f94_d4b17deb3b68.slice/crio-f9322c1297cd50064d6501f3307a3f60923e5283781b3aee465894fb38adfc37 WatchSource:0}: Error finding container f9322c1297cd50064d6501f3307a3f60923e5283781b3aee465894fb38adfc37: Status 404 returned error can't find the container with id f9322c1297cd50064d6501f3307a3f60923e5283781b3aee465894fb38adfc37
Sep 18 07:00:27.629839 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.629736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" event={"ID":"84176077-bde3-4204-8f7a-c54e40574744","Type":"ContainerStarted","Data":"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"}
Sep 18 07:00:27.629839 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.629806    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" event={"ID":"84176077-bde3-4204-8f7a-c54e40574744","Type":"ContainerStarted","Data":"82efc394ab1fbc1afd542bc7f50b597cb1bfd210330513fb2e4764876cceb6b3"}
Sep 18 07:00:27.629839 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.629836    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:27.631359 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.631336    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" event={"ID":"566c6592-1f21-4e44-9f94-d4b17deb3b68","Type":"ContainerStarted","Data":"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"}
Sep 18 07:00:27.631482 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.631381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" event={"ID":"566c6592-1f21-4e44-9f94-d4b17deb3b68","Type":"ContainerStarted","Data":"f9322c1297cd50064d6501f3307a3f60923e5283781b3aee465894fb38adfc37"}
Sep 18 07:00:27.631528 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.631474    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:00:27.646773 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.646727    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" podStartSLOduration=1.224292659 podStartE2EDuration="1.646712401s" totalImagesPullingTime="422.419742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:27.644564533 +0000 UTC m=+2307.227950187" watchObservedRunningTime="2026-09-18 07:00:27.646712401 +0000 UTC m=+2307.230098053"
Sep 18 07:00:27.661164 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:27.661104    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" podStartSLOduration=1.6610848900000001 podStartE2EDuration="1.66108489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:00:27.658907675 +0000 UTC m=+2307.242293329" watchObservedRunningTime="2026-09-18 07:00:27.66108489 +0000 UTC m=+2307.244470544"
Sep 18 07:00:33.642356 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:33.642321    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:00:38.639129 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:00:38.639095    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:01:04.837313 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:04.837221    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:01:04.837881 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:04.837424    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" podUID="84176077-bde3-4204-8f7a-c54e40574744" containerName="maas-api" containerID="cri-o://3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9" gracePeriod=30
Sep 18 07:01:04.948075 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:04.948041    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:01:04.948286 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:04.948263    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" podUID="566c6592-1f21-4e44-9f94-d4b17deb3b68" containerName="payload-pre-processing" containerID="cri-o://196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267" gracePeriod=30
Sep 18 07:01:05.109375 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.109348    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:01:05.183978 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.183944    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs\") pod \"84176077-bde3-4204-8f7a-c54e40574744\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") "
Sep 18 07:01:05.184168 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.183987    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nql7d\" (UniqueName: \"kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d\") pod \"84176077-bde3-4204-8f7a-c54e40574744\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") "
Sep 18 07:01:05.184168 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.184056    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca\") pod \"84176077-bde3-4204-8f7a-c54e40574744\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") "
Sep 18 07:01:05.184295 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.184202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls\") pod \"84176077-bde3-4204-8f7a-c54e40574744\" (UID: \"84176077-bde3-4204-8f7a-c54e40574744\") "
Sep 18 07:01:05.184476 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.184443    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84176077-bde3-4204-8f7a-c54e40574744" (UID: "84176077-bde3-4204-8f7a-c54e40574744"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:01:05.184624 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.184608    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84176077-bde3-4204-8f7a-c54e40574744-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.187756 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.187730    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d" (OuterVolumeSpecName: "kube-api-access-nql7d") pod "84176077-bde3-4204-8f7a-c54e40574744" (UID: "84176077-bde3-4204-8f7a-c54e40574744"). InnerVolumeSpecName "kube-api-access-nql7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:01:05.187900 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.187873    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84176077-bde3-4204-8f7a-c54e40574744" (UID: "84176077-bde3-4204-8f7a-c54e40574744"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:01:05.187973 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.187931    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84176077-bde3-4204-8f7a-c54e40574744" (UID: "84176077-bde3-4204-8f7a-c54e40574744"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:01:05.210100 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.210072    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:01:05.285702 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.285664    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume\") pod \"566c6592-1f21-4e44-9f94-d4b17deb3b68\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") "
Sep 18 07:01:05.285873 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.285735    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dgt5\" (UniqueName: \"kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5\") pod \"566c6592-1f21-4e44-9f94-d4b17deb3b68\" (UID: \"566c6592-1f21-4e44-9f94-d4b17deb3b68\") "
Sep 18 07:01:05.285916 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.285903    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.285952 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.285917    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nql7d\" (UniqueName: \"kubernetes.io/projected/84176077-bde3-4204-8f7a-c54e40574744-kube-api-access-nql7d\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.285952 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.285928    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84176077-bde3-4204-8f7a-c54e40574744-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.286039 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.286016    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "566c6592-1f21-4e44-9f94-d4b17deb3b68" (UID: "566c6592-1f21-4e44-9f94-d4b17deb3b68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:01:05.287766 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.287739    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5" (OuterVolumeSpecName: "kube-api-access-4dgt5") pod "566c6592-1f21-4e44-9f94-d4b17deb3b68" (UID: "566c6592-1f21-4e44-9f94-d4b17deb3b68"). InnerVolumeSpecName "kube-api-access-4dgt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:01:05.386593 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.386509    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/566c6592-1f21-4e44-9f94-d4b17deb3b68-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.386593 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.386537    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgt5\" (UniqueName: \"kubernetes.io/projected/566c6592-1f21-4e44-9f94-d4b17deb3b68-kube-api-access-4dgt5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:05.827984 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.827945    2567 generic.go:356] "Generic (PLEG): container finished" podID="566c6592-1f21-4e44-9f94-d4b17deb3b68" containerID="196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267" exitCode=0
Sep 18 07:01:05.828197 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.827994    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"
Sep 18 07:01:05.828197 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.828027    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" event={"ID":"566c6592-1f21-4e44-9f94-d4b17deb3b68","Type":"ContainerDied","Data":"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"}
Sep 18 07:01:05.828197 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.828082    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26" event={"ID":"566c6592-1f21-4e44-9f94-d4b17deb3b68","Type":"ContainerDied","Data":"f9322c1297cd50064d6501f3307a3f60923e5283781b3aee465894fb38adfc37"}
Sep 18 07:01:05.828197 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.828116    2567 scope.go:117] "RemoveContainer" containerID="196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"
Sep 18 07:01:05.829573 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.829550    2567 generic.go:356] "Generic (PLEG): container finished" podID="84176077-bde3-4204-8f7a-c54e40574744" containerID="3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9" exitCode=0
Sep 18 07:01:05.829682 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.829580    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" event={"ID":"84176077-bde3-4204-8f7a-c54e40574744","Type":"ContainerDied","Data":"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"}
Sep 18 07:01:05.829682 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.829605    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"
Sep 18 07:01:05.829682 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.829624    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl" event={"ID":"84176077-bde3-4204-8f7a-c54e40574744","Type":"ContainerDied","Data":"82efc394ab1fbc1afd542bc7f50b597cb1bfd210330513fb2e4764876cceb6b3"}
Sep 18 07:01:05.838424 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.838253    2567 scope.go:117] "RemoveContainer" containerID="196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"
Sep 18 07:01:05.838680 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:01:05.838549    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267\": container with ID starting with 196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267 not found: ID does not exist" containerID="196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"
Sep 18 07:01:05.838680 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.838576    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267"} err="failed to get container status \"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267\": rpc error: code = NotFound desc = could not find container \"196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267\": container with ID starting with 196efb94eda20e82aaf711aa024d5e4b5a655ab699c9d9f43b009dff50e25267 not found: ID does not exist"
Sep 18 07:01:05.838680 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.838597    2567 scope.go:117] "RemoveContainer" containerID="3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"
Sep 18 07:01:05.848864 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.848844    2567 scope.go:117] "RemoveContainer" containerID="3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"
Sep 18 07:01:05.849378 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:01:05.849358    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9\": container with ID starting with 3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9 not found: ID does not exist" containerID="3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"
Sep 18 07:01:05.849460 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.849386    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9"} err="failed to get container status \"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9\": rpc error: code = NotFound desc = could not find container \"3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9\": container with ID starting with 3ead2063ac451068028013e088514bf8856383f1e565e0f091728f8c0ca0f7a9 not found: ID does not exist"
Sep 18 07:01:05.853296 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.853188    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:01:05.855275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.855254    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-75ce90-967b4c555-bwkfl"]
Sep 18 07:01:05.864006 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.863985    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:01:05.867657 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:05.867636    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-75ce90-58595b76f8-ljc26"]
Sep 18 07:01:06.909513 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:06.909478    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566c6592-1f21-4e44-9f94-d4b17deb3b68" path="/var/lib/kubelet/pods/566c6592-1f21-4e44-9f94-d4b17deb3b68/volumes"
Sep 18 07:01:06.909900 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:06.909885    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84176077-bde3-4204-8f7a-c54e40574744" path="/var/lib/kubelet/pods/84176077-bde3-4204-8f7a-c54e40574744/volumes"
Sep 18 07:01:30.313156 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.313107    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 07:01:30.313526 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.313341    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" podUID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" containerName="maas-api" containerID="cri-o://c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938" gracePeriod=30
Sep 18 07:01:30.562441 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.562412    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 07:01:30.712909 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.712878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca\") pod \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") "
Sep 18 07:01:30.713103 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.712940    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") pod \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") "
Sep 18 07:01:30.713103 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.712997    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tl75\" (UniqueName: \"kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75\") pod \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") "
Sep 18 07:01:30.713103 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.713036    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs\") pod \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\" (UID: \"3e0afe96-74f3-4135-8e83-12c8ef2fc13d\") "
Sep 18 07:01:30.713291 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.713263    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e0afe96-74f3-4135-8e83-12c8ef2fc13d" (UID: "3e0afe96-74f3-4135-8e83-12c8ef2fc13d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:01:30.713417 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.713399    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:30.715123 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.715090    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e0afe96-74f3-4135-8e83-12c8ef2fc13d" (UID: "3e0afe96-74f3-4135-8e83-12c8ef2fc13d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:01:30.715356 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.715180    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e0afe96-74f3-4135-8e83-12c8ef2fc13d" (UID: "3e0afe96-74f3-4135-8e83-12c8ef2fc13d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:01:30.715356 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.715212    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75" (OuterVolumeSpecName: "kube-api-access-4tl75") pod "3e0afe96-74f3-4135-8e83-12c8ef2fc13d" (UID: "3e0afe96-74f3-4135-8e83-12c8ef2fc13d"). InnerVolumeSpecName "kube-api-access-4tl75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:01:30.814746 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.814711    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:30.814746 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.814739    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tl75\" (UniqueName: \"kubernetes.io/projected/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-kube-api-access-4tl75\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:30.814746 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.814750    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e0afe96-74f3-4135-8e83-12c8ef2fc13d-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:01:30.953233 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.953200    2567 generic.go:356] "Generic (PLEG): container finished" podID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" containerID="c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938" exitCode=0
Sep 18 07:01:30.953381 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.953238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" event={"ID":"3e0afe96-74f3-4135-8e83-12c8ef2fc13d","Type":"ContainerDied","Data":"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"}
Sep 18 07:01:30.953381 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.953264    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"
Sep 18 07:01:30.953381 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.953279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x" event={"ID":"3e0afe96-74f3-4135-8e83-12c8ef2fc13d","Type":"ContainerDied","Data":"1eb205e8a225d3e70ad9714e2b28e5a2d34f6ad932666be49a7bf8d00742e5e4"}
Sep 18 07:01:30.953381 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.953300    2567 scope.go:117] "RemoveContainer" containerID="c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"
Sep 18 07:01:30.962770 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.962752    2567 scope.go:117] "RemoveContainer" containerID="c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"
Sep 18 07:01:30.963068 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:01:30.963025    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938\": container with ID starting with c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938 not found: ID does not exist" containerID="c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"
Sep 18 07:01:30.963068 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.963051    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938"} err="failed to get container status \"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938\": rpc error: code = NotFound desc = could not find container \"c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938\": container with ID starting with c358ea1e911c86abeff0d6e407c18d07c3c532339897f802891b8016998a8938 not found: ID does not exist"
Sep 18 07:01:30.971034 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.971012    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 07:01:30.974708 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:30.974688    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7bd58-598d56654b-crk8x"]
Sep 18 07:01:32.909559 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:01:32.909518    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" path="/var/lib/kubelet/pods/3e0afe96-74f3-4135-8e83-12c8ef2fc13d/volumes"
Sep 18 07:02:01.016714 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:01.016684    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 07:02:01.023984 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:01.023962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 07:02:36.505866 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.505796    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 07:02:36.506662 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.506033    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" podUID="531fe294-1340-42f6-884e-ec15bf20d614" containerName="maas-api" containerID="cri-o://6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0" gracePeriod=30
Sep 18 07:02:36.553403 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.553357    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 07:02:36.553897 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.553841    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" podUID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" containerName="payload-processing" containerID="cri-o://30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c" gracePeriod=30
Sep 18 07:02:36.562954 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.562903    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 07:02:36.563360 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.563306    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" podUID="a25ef891-9846-447c-825f-501808accd4f" containerName="payload-pre-processing" containerID="cri-o://03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1" gracePeriod=30
Sep 18 07:02:36.832035 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.832009    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 07:02:36.859631 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.859601    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 07:02:36.869729 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.869697    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 07:02:36.920631 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920602    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrpq7\" (UniqueName: \"kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7\") pod \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") "
Sep 18 07:02:36.920631 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920633    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtfzc\" (UniqueName: \"kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc\") pod \"a25ef891-9846-447c-825f-501808accd4f\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") "
Sep 18 07:02:36.920837 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920662    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls\") pod \"531fe294-1340-42f6-884e-ec15bf20d614\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") "
Sep 18 07:02:36.920837 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920690    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume\") pod \"a25ef891-9846-447c-825f-501808accd4f\" (UID: \"a25ef891-9846-447c-825f-501808accd4f\") "
Sep 18 07:02:36.920837 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920717    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume\") pod \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\" (UID: \"d430db3b-2a38-40fe-8dda-cce3fe84aa96\") "
Sep 18 07:02:36.920837 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920831    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs\") pod \"531fe294-1340-42f6-884e-ec15bf20d614\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") "
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920855    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca\") pod \"531fe294-1340-42f6-884e-ec15bf20d614\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") "
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.920907    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5d8j\" (UniqueName: \"kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j\") pod \"531fe294-1340-42f6-884e-ec15bf20d614\" (UID: \"531fe294-1340-42f6-884e-ec15bf20d614\") "
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.921123    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a25ef891-9846-447c-825f-501808accd4f" (UID: "a25ef891-9846-447c-825f-501808accd4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.921175    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d430db3b-2a38-40fe-8dda-cce3fe84aa96" (UID: "d430db3b-2a38-40fe-8dda-cce3fe84aa96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.921252    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a25ef891-9846-447c-825f-501808accd4f-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:36.921275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.921272    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d430db3b-2a38-40fe-8dda-cce3fe84aa96-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:36.921642 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.921618    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "531fe294-1340-42f6-884e-ec15bf20d614" (UID: "531fe294-1340-42f6-884e-ec15bf20d614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:02:36.922979 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.922955    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "531fe294-1340-42f6-884e-ec15bf20d614" (UID: "531fe294-1340-42f6-884e-ec15bf20d614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:02:36.923357 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.923334    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7" (OuterVolumeSpecName: "kube-api-access-vrpq7") pod "d430db3b-2a38-40fe-8dda-cce3fe84aa96" (UID: "d430db3b-2a38-40fe-8dda-cce3fe84aa96"). InnerVolumeSpecName "kube-api-access-vrpq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:02:36.923464 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.923356    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc" (OuterVolumeSpecName: "kube-api-access-dtfzc") pod "a25ef891-9846-447c-825f-501808accd4f" (UID: "a25ef891-9846-447c-825f-501808accd4f"). InnerVolumeSpecName "kube-api-access-dtfzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:02:36.923464 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.923369    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "531fe294-1340-42f6-884e-ec15bf20d614" (UID: "531fe294-1340-42f6-884e-ec15bf20d614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:02:36.923464 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:36.923407    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j" (OuterVolumeSpecName: "kube-api-access-d5d8j") pod "531fe294-1340-42f6-884e-ec15bf20d614" (UID: "531fe294-1340-42f6-884e-ec15bf20d614"). InnerVolumeSpecName "kube-api-access-d5d8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:02:37.022845 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022751    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpq7\" (UniqueName: \"kubernetes.io/projected/d430db3b-2a38-40fe-8dda-cce3fe84aa96-kube-api-access-vrpq7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.022845 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022789    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfzc\" (UniqueName: \"kubernetes.io/projected/a25ef891-9846-447c-825f-501808accd4f-kube-api-access-dtfzc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.022845 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022807    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.022845 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022821    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531fe294-1340-42f6-884e-ec15bf20d614-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.022845 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022844    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531fe294-1340-42f6-884e-ec15bf20d614-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.024255 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.022858    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5d8j\" (UniqueName: \"kubernetes.io/projected/531fe294-1340-42f6-884e-ec15bf20d614-kube-api-access-d5d8j\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:37.279023 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.278936    2567 generic.go:356] "Generic (PLEG): container finished" podID="531fe294-1340-42f6-884e-ec15bf20d614" containerID="6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0" exitCode=0
Sep 18 07:02:37.279023 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.278982    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"
Sep 18 07:02:37.279256 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.279030    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" event={"ID":"531fe294-1340-42f6-884e-ec15bf20d614","Type":"ContainerDied","Data":"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"}
Sep 18 07:02:37.279256 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.279185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql" event={"ID":"531fe294-1340-42f6-884e-ec15bf20d614","Type":"ContainerDied","Data":"f135ad3c3fc09381c06b6f56f8ec4cc90dc1368148c6f911534e71def26fccac"}
Sep 18 07:02:37.279256 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.279220    2567 scope.go:117] "RemoveContainer" containerID="6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"
Sep 18 07:02:37.280510 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.280486    2567 generic.go:356] "Generic (PLEG): container finished" podID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" containerID="30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c" exitCode=0
Sep 18 07:02:37.280596 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.280532    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"
Sep 18 07:02:37.280596 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.280564    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" event={"ID":"d430db3b-2a38-40fe-8dda-cce3fe84aa96","Type":"ContainerDied","Data":"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"}
Sep 18 07:02:37.280701 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.280609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc" event={"ID":"d430db3b-2a38-40fe-8dda-cce3fe84aa96","Type":"ContainerDied","Data":"4de66fb6e75b8aa66869eb717c573b3bc87a2e11ff3e2f0eb2b6ae2e2f65a51e"}
Sep 18 07:02:37.282244 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.282219    2567 generic.go:356] "Generic (PLEG): container finished" podID="a25ef891-9846-447c-825f-501808accd4f" containerID="03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1" exitCode=0
Sep 18 07:02:37.282370 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.282253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" event={"ID":"a25ef891-9846-447c-825f-501808accd4f","Type":"ContainerDied","Data":"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"}
Sep 18 07:02:37.282370 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.282278    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"
Sep 18 07:02:37.282370 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.282293    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk" event={"ID":"a25ef891-9846-447c-825f-501808accd4f","Type":"ContainerDied","Data":"41d3abd84817c9684ca32fa7b34facf1af440bc6e22eab826251728994f1273c"}
Sep 18 07:02:37.288948 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.288929    2567 scope.go:117] "RemoveContainer" containerID="6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"
Sep 18 07:02:37.289280 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:02:37.289254    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0\": container with ID starting with 6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0 not found: ID does not exist" containerID="6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"
Sep 18 07:02:37.289400 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.289286    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0"} err="failed to get container status \"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0\": rpc error: code = NotFound desc = could not find container \"6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0\": container with ID starting with 6009554bf94aefb446d46e3e1360bd8e8b6ed4b66f9298780e933a382213ddb0 not found: ID does not exist"
Sep 18 07:02:37.289400 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.289303    2567 scope.go:117] "RemoveContainer" containerID="30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"
Sep 18 07:02:37.298568 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.298552    2567 scope.go:117] "RemoveContainer" containerID="30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"
Sep 18 07:02:37.298832 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:02:37.298814    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c\": container with ID starting with 30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c not found: ID does not exist" containerID="30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"
Sep 18 07:02:37.298876 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.298839    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c"} err="failed to get container status \"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c\": rpc error: code = NotFound desc = could not find container \"30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c\": container with ID starting with 30faa89c0d6e31fae77d97ef886573f55937fc60868e3caad35356dbcc881e5c not found: ID does not exist"
Sep 18 07:02:37.298876 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.298855    2567 scope.go:117] "RemoveContainer" containerID="03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"
Sep 18 07:02:37.305593 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.305568    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 07:02:37.306433 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.306417    2567 scope.go:117] "RemoveContainer" containerID="03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"
Sep 18 07:02:37.306688 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:02:37.306666    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1\": container with ID starting with 03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1 not found: ID does not exist" containerID="03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"
Sep 18 07:02:37.306782 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.306696    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1"} err="failed to get container status \"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1\": rpc error: code = NotFound desc = could not find container \"03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1\": container with ID starting with 03f3ab0fc8030cfad3ef2e48f02600a9346f48487a51ded6a1252bccd9c30bd1 not found: ID does not exist"
Sep 18 07:02:37.309369 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.309349    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ec2077-55c9ddbc74-bhwql"]
Sep 18 07:02:37.319027 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.319006    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 07:02:37.322684 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.322663    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ec2077-977d8c666-h6jvk"]
Sep 18 07:02:37.332906 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.332886    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 07:02:37.339062 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:37.339040    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ec2077-99f46f5c9-d9jtc"]
Sep 18 07:02:38.916013 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:38.915979    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531fe294-1340-42f6-884e-ec15bf20d614" path="/var/lib/kubelet/pods/531fe294-1340-42f6-884e-ec15bf20d614/volumes"
Sep 18 07:02:38.916403 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:38.916360    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25ef891-9846-447c-825f-501808accd4f" path="/var/lib/kubelet/pods/a25ef891-9846-447c-825f-501808accd4f/volumes"
Sep 18 07:02:38.916674 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:38.916661    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" path="/var/lib/kubelet/pods/d430db3b-2a38-40fe-8dda-cce3fe84aa96/volumes"
Sep 18 07:02:53.989956 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:53.989918    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 07:02:53.990458 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:53.990193    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" podUID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" containerName="maas-api" containerID="cri-o://3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef" gracePeriod=30
Sep 18 07:02:54.238405 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.238378    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 07:02:54.375004 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.374910    2567 generic.go:356] "Generic (PLEG): container finished" podID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" containerID="3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef" exitCode=0
Sep 18 07:02:54.375004 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.374971    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"
Sep 18 07:02:54.375004 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.374993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" event={"ID":"0e12cdfb-19c5-4057-b810-8f0559b73fc3","Type":"ContainerDied","Data":"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"}
Sep 18 07:02:54.375253 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.375052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw" event={"ID":"0e12cdfb-19c5-4057-b810-8f0559b73fc3","Type":"ContainerDied","Data":"99fcad99204dd66011fc62fbf342de3f1447a4a65802c20d63bd68032b419920"}
Sep 18 07:02:54.375253 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.375067    2567 scope.go:117] "RemoveContainer" containerID="3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"
Sep 18 07:02:54.383280 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383252    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca\") pod \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") "
Sep 18 07:02:54.383412 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383304    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tswc9\" (UniqueName: \"kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9\") pod \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") "
Sep 18 07:02:54.383412 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383372    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs\") pod \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") "
Sep 18 07:02:54.383536 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383495    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls\") pod \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\" (UID: \"0e12cdfb-19c5-4057-b810-8f0559b73fc3\") "
Sep 18 07:02:54.383626 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383601    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e12cdfb-19c5-4057-b810-8f0559b73fc3" (UID: "0e12cdfb-19c5-4057-b810-8f0559b73fc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:02:54.383858 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.383827    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e12cdfb-19c5-4057-b810-8f0559b73fc3-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:54.384174 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.384154    2567 scope.go:117] "RemoveContainer" containerID="3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"
Sep 18 07:02:54.384451 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:02:54.384425    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef\": container with ID starting with 3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef not found: ID does not exist" containerID="3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"
Sep 18 07:02:54.384541 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.384459    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef"} err="failed to get container status \"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef\": rpc error: code = NotFound desc = could not find container \"3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef\": container with ID starting with 3cf381645b4ee7887c6013efbaa08fa12d5bfba9db4fd6ceb22bd5a0595809ef not found: ID does not exist"
Sep 18 07:02:54.385521 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.385499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e12cdfb-19c5-4057-b810-8f0559b73fc3" (UID: "0e12cdfb-19c5-4057-b810-8f0559b73fc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:02:54.385606 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.385527    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e12cdfb-19c5-4057-b810-8f0559b73fc3" (UID: "0e12cdfb-19c5-4057-b810-8f0559b73fc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:02:54.385606 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.385547    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9" (OuterVolumeSpecName: "kube-api-access-tswc9") pod "0e12cdfb-19c5-4057-b810-8f0559b73fc3" (UID: "0e12cdfb-19c5-4057-b810-8f0559b73fc3"). InnerVolumeSpecName "kube-api-access-tswc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:02:54.485217 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.485186    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tswc9\" (UniqueName: \"kubernetes.io/projected/0e12cdfb-19c5-4057-b810-8f0559b73fc3-kube-api-access-tswc9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:54.485217 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.485213    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:54.485217 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.485225    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e12cdfb-19c5-4057-b810-8f0559b73fc3-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:02:54.698527 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.698457    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 07:02:54.700705 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.700680    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8fdd5b-676959588b-82vmw"]
Sep 18 07:02:54.910120 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:02:54.910087    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" path="/var/lib/kubelet/pods/0e12cdfb-19c5-4057-b810-8f0559b73fc3/volumes"
Sep 18 07:03:03.672971 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.672932    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 07:03:03.673460 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.673213    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" podUID="555fa4e1-7fbb-4a94-b788-cfa495d55553" containerName="manager" containerID="cri-o://ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc" gracePeriod=30
Sep 18 07:03:03.921022 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.920995    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 07:03:03.968909 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.968872    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert\") pod \"555fa4e1-7fbb-4a94-b788-cfa495d55553\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") "
Sep 18 07:03:03.969086 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.968917    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs\") pod \"555fa4e1-7fbb-4a94-b788-cfa495d55553\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") "
Sep 18 07:03:03.969086 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.968943    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvfb7\" (UniqueName: \"kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7\") pod \"555fa4e1-7fbb-4a94-b788-cfa495d55553\" (UID: \"555fa4e1-7fbb-4a94-b788-cfa495d55553\") "
Sep 18 07:03:03.970958 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.970928    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "555fa4e1-7fbb-4a94-b788-cfa495d55553" (UID: "555fa4e1-7fbb-4a94-b788-cfa495d55553"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:03:03.971058 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.970980    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "555fa4e1-7fbb-4a94-b788-cfa495d55553" (UID: "555fa4e1-7fbb-4a94-b788-cfa495d55553"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:03:03.971058 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:03.970991    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7" (OuterVolumeSpecName: "kube-api-access-bvfb7") pod "555fa4e1-7fbb-4a94-b788-cfa495d55553" (UID: "555fa4e1-7fbb-4a94-b788-cfa495d55553"). InnerVolumeSpecName "kube-api-access-bvfb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:03:04.070326 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.070291    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-webhook-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:03:04.070326 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.070317    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/555fa4e1-7fbb-4a94-b788-cfa495d55553-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:03:04.070326 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.070327    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfb7\" (UniqueName: \"kubernetes.io/projected/555fa4e1-7fbb-4a94-b788-cfa495d55553-kube-api-access-bvfb7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:03:04.425811 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.425708    2567 generic.go:356] "Generic (PLEG): container finished" podID="555fa4e1-7fbb-4a94-b788-cfa495d55553" containerID="ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc" exitCode=0
Sep 18 07:03:04.426106 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.426085    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" event={"ID":"555fa4e1-7fbb-4a94-b788-cfa495d55553","Type":"ContainerDied","Data":"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"}
Sep 18 07:03:04.426335 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.426318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb" event={"ID":"555fa4e1-7fbb-4a94-b788-cfa495d55553","Type":"ContainerDied","Data":"824eeca29c4c55e89a560fa83c75a1b3031a68e492332a4eb853282442a5963a"}
Sep 18 07:03:04.426466 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.426453    2567 scope.go:117] "RemoveContainer" containerID="ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"
Sep 18 07:03:04.426759 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.426742    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-v9dlb"
Sep 18 07:03:04.441642 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.441622    2567 scope.go:117] "RemoveContainer" containerID="ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"
Sep 18 07:03:04.441923 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:03:04.441902    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc\": container with ID starting with ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc not found: ID does not exist" containerID="ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"
Sep 18 07:03:04.441992 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.441935    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc"} err="failed to get container status \"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc\": rpc error: code = NotFound desc = could not find container \"ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc\": container with ID starting with ab31571632cf2b2e43ef6c60fc12a82d22a1b82cc72835e967832fc9c22154fc not found: ID does not exist"
Sep 18 07:03:04.451228 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.451105    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 07:03:04.452609 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.452589    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-v9dlb"]
Sep 18 07:03:04.908994 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:03:04.908955    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555fa4e1-7fbb-4a94-b788-cfa495d55553" path="/var/lib/kubelet/pods/555fa4e1-7fbb-4a94-b788-cfa495d55553/volumes"
Sep 18 07:04:05.216313 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216278    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216624    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566c6592-1f21-4e44-9f94-d4b17deb3b68" containerName="payload-pre-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216635    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="566c6592-1f21-4e44-9f94-d4b17deb3b68" containerName="payload-pre-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216652    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25ef891-9846-447c-825f-501808accd4f" containerName="payload-pre-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216657    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25ef891-9846-447c-825f-501808accd4f" containerName="payload-pre-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216666    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" containerName="payload-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216672    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" containerName="payload-processing"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216678    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84176077-bde3-4204-8f7a-c54e40574744" containerName="maas-api"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216683    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="84176077-bde3-4204-8f7a-c54e40574744" containerName="maas-api"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216694    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531fe294-1340-42f6-884e-ec15bf20d614" containerName="maas-api"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216699    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="531fe294-1340-42f6-884e-ec15bf20d614" containerName="maas-api"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216712    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555fa4e1-7fbb-4a94-b788-cfa495d55553" containerName="manager"
Sep 18 07:04:05.216716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216717    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="555fa4e1-7fbb-4a94-b788-cfa495d55553" containerName="manager"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216723    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216730    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216736    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216741    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216791    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="566c6592-1f21-4e44-9f94-d4b17deb3b68" containerName="payload-pre-processing"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216798    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="84176077-bde3-4204-8f7a-c54e40574744" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216804    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="531fe294-1340-42f6-884e-ec15bf20d614" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216810    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25ef891-9846-447c-825f-501808accd4f" containerName="payload-pre-processing"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216815    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0afe96-74f3-4135-8e83-12c8ef2fc13d" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216822    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d430db3b-2a38-40fe-8dda-cce3fe84aa96" containerName="payload-processing"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216827    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e12cdfb-19c5-4057-b810-8f0559b73fc3" containerName="maas-api"
Sep 18 07:04:05.217097 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.216832    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="555fa4e1-7fbb-4a94-b788-cfa495d55553" containerName="manager"
Sep 18 07:04:05.220531 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.220514    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.224094 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.224071    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 07:04:05.224094 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.224086    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b28rl\""
Sep 18 07:04:05.224290 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.224101    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 07:04:05.224290 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.224114    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 07:04:05.230066 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.230041    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:05.306881 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.306845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.307069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.306913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.307069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.306947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psr5j\" (UniqueName: \"kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.407674 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.407644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.407867 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.407681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psr5j\" (UniqueName: \"kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.407867 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.407737    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.410061 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.410032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.410214 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.410107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.415904 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.415884    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psr5j\" (UniqueName: \"kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j\") pod \"maas-controller-84c8b64bc5-gt82d\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") " pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.532494 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.532404    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:05.801260 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.801236    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:05.803362 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:04:05.803337    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97481f74_1a6f_4316_875b_816c5bbeae8a.slice/crio-b26971a470389549d2e90e303293b8618ff7d571be8bbda6fea956cd86268e39 WatchSource:0}: Error finding container b26971a470389549d2e90e303293b8618ff7d571be8bbda6fea956cd86268e39: Status 404 returned error can't find the container with id b26971a470389549d2e90e303293b8618ff7d571be8bbda6fea956cd86268e39
Sep 18 07:04:05.804812 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:05.804788    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 07:04:06.711487 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:06.711450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" event={"ID":"97481f74-1a6f-4316-875b-816c5bbeae8a","Type":"ContainerStarted","Data":"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"}
Sep 18 07:04:06.711957 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:06.711509    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" event={"ID":"97481f74-1a6f-4316-875b-816c5bbeae8a","Type":"ContainerStarted","Data":"b26971a470389549d2e90e303293b8618ff7d571be8bbda6fea956cd86268e39"}
Sep 18 07:04:06.711957 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:06.711561    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:06.729331 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:06.729281    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" podStartSLOduration=1.134064467 podStartE2EDuration="1.729265133s" totalImagesPullingTime="595.200666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:04:06.72709394 +0000 UTC m=+2526.310479604" watchObservedRunningTime="2026-09-18 07:04:06.729265133 +0000 UTC m=+2526.312650832"
Sep 18 07:04:17.718869 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:17.718837    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:27.103275 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.103236    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:27.103788 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.103448    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" podUID="97481f74-1a6f-4316-875b-816c5bbeae8a" containerName="manager" containerID="cri-o://cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d" gracePeriod=30
Sep 18 07:04:27.352069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.352045    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:27.509520 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.509485    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psr5j\" (UniqueName: \"kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j\") pod \"97481f74-1a6f-4316-875b-816c5bbeae8a\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") "
Sep 18 07:04:27.509716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.509549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs\") pod \"97481f74-1a6f-4316-875b-816c5bbeae8a\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") "
Sep 18 07:04:27.509716 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.509666    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert\") pod \"97481f74-1a6f-4316-875b-816c5bbeae8a\" (UID: \"97481f74-1a6f-4316-875b-816c5bbeae8a\") "
Sep 18 07:04:27.511762 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.511726    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j" (OuterVolumeSpecName: "kube-api-access-psr5j") pod "97481f74-1a6f-4316-875b-816c5bbeae8a" (UID: "97481f74-1a6f-4316-875b-816c5bbeae8a"). InnerVolumeSpecName "kube-api-access-psr5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:04:27.511889 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.511769    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "97481f74-1a6f-4316-875b-816c5bbeae8a" (UID: "97481f74-1a6f-4316-875b-816c5bbeae8a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:04:27.511889 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.511790    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97481f74-1a6f-4316-875b-816c5bbeae8a" (UID: "97481f74-1a6f-4316-875b-816c5bbeae8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:04:27.611090 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.611055    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:04:27.611090 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.611082    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97481f74-1a6f-4316-875b-816c5bbeae8a-webhook-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:04:27.611090 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.611091    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psr5j\" (UniqueName: \"kubernetes.io/projected/97481f74-1a6f-4316-875b-816c5bbeae8a-kube-api-access-psr5j\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:04:27.808389 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.808304    2567 generic.go:356] "Generic (PLEG): container finished" podID="97481f74-1a6f-4316-875b-816c5bbeae8a" containerID="cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d" exitCode=0
Sep 18 07:04:27.808389 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.808355    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-gt82d"
Sep 18 07:04:27.808389 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.808369    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" event={"ID":"97481f74-1a6f-4316-875b-816c5bbeae8a","Type":"ContainerDied","Data":"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"}
Sep 18 07:04:27.808627 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.808413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-gt82d" event={"ID":"97481f74-1a6f-4316-875b-816c5bbeae8a","Type":"ContainerDied","Data":"b26971a470389549d2e90e303293b8618ff7d571be8bbda6fea956cd86268e39"}
Sep 18 07:04:27.808627 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.808435    2567 scope.go:117] "RemoveContainer" containerID="cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"
Sep 18 07:04:27.817297 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.817278    2567 scope.go:117] "RemoveContainer" containerID="cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"
Sep 18 07:04:27.817549 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:04:27.817530    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d\": container with ID starting with cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d not found: ID does not exist" containerID="cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"
Sep 18 07:04:27.817604 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.817556    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d"} err="failed to get container status \"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d\": rpc error: code = NotFound desc = could not find container \"cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d\": container with ID starting with cad3db391e739a9e8a95d0a31d326d9bbdc4cdcf7fb57ca5f7d73aa7a64a8d2d not found: ID does not exist"
Sep 18 07:04:27.829702 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.829680    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:27.832975 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:27.832949    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-gt82d"]
Sep 18 07:04:28.608522 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.608486    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-lsx4f"]
Sep 18 07:04:28.608915 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.608899    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97481f74-1a6f-4316-875b-816c5bbeae8a" containerName="manager"
Sep 18 07:04:28.608964 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.608918    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="97481f74-1a6f-4316-875b-816c5bbeae8a" containerName="manager"
Sep 18 07:04:28.609009 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.608999    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="97481f74-1a6f-4316-875b-816c5bbeae8a" containerName="manager"
Sep 18 07:04:28.613591 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.613565    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.616557 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.616538    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 07:04:28.617779 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.617749    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 07:04:28.617779 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.617768    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b28rl\""
Sep 18 07:04:28.617943 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.617824    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 07:04:28.622734 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.622708    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-lsx4f"]
Sep 18 07:04:28.720581 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.720539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkjv\" (UniqueName: \"kubernetes.io/projected/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-kube-api-access-9zkjv\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.720759 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.720588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-metrics-certs\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.720759 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.720615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-webhook-cert\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.821777 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.821742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-metrics-certs\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.821777 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.821780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-webhook-cert\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.821989 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.821848    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkjv\" (UniqueName: \"kubernetes.io/projected/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-kube-api-access-9zkjv\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.824268 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.824243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-webhook-cert\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.824384 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.824334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-metrics-certs\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.829150 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.829117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkjv\" (UniqueName: \"kubernetes.io/projected/cd896cb4-8a74-4b33-9f6f-0d8b90de6700-kube-api-access-9zkjv\") pod \"maas-controller-84c8b64bc5-lsx4f\" (UID: \"cd896cb4-8a74-4b33-9f6f-0d8b90de6700\") " pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:28.908920 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.908845    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97481f74-1a6f-4316-875b-816c5bbeae8a" path="/var/lib/kubelet/pods/97481f74-1a6f-4316-875b-816c5bbeae8a/volumes"
Sep 18 07:04:28.926242 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:28.926211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:29.195436 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:29.195410    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84c8b64bc5-lsx4f"]
Sep 18 07:04:29.821111 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:29.821072    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f" event={"ID":"cd896cb4-8a74-4b33-9f6f-0d8b90de6700","Type":"ContainerStarted","Data":"2b1a435a2d5374b27c650733ba90c342bf3d0458bb54af85f91d8287684fec89"}
Sep 18 07:04:29.821524 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:29.821186    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f" event={"ID":"cd896cb4-8a74-4b33-9f6f-0d8b90de6700","Type":"ContainerStarted","Data":"c1f9343a53064906f6b656d97f1559b6c45db01203d0bdf38dfdd82021588227"}
Sep 18 07:04:29.821524 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:29.821218    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:04:29.839435 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:29.839351    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f" podStartSLOduration=1.4633846560000001 podStartE2EDuration="1.839336652s" totalImagesPullingTime="375.951996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:04:29.83665447 +0000 UTC m=+2549.420040124" watchObservedRunningTime="2026-09-18 07:04:29.839336652 +0000 UTC m=+2549.422722304"
Sep 18 07:04:40.827441 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:04:40.827407    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84c8b64bc5-lsx4f"
Sep 18 07:07:01.055338 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:07:01.055267    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 07:07:01.062323 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:07:01.062299    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 07:08:50.955897 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:50.955863    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"]
Sep 18 07:08:50.959268 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:50.959242    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:50.973655 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:50.973626    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"]
Sep 18 07:08:51.058450 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.058404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-maas-api-tls\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.058450 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.058444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv44j\" (UniqueName: \"kubernetes.io/projected/c42a6620-3692-4055-8497-6826b9bb063e-kube-api-access-lv44j\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.058717 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.058535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42a6620-3692-4055-8497-6826b9bb063e-openshift-service-ca\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.058717 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.058605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-metrics-certs\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.159043 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.159007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-maas-api-tls\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.159043 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.159043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv44j\" (UniqueName: \"kubernetes.io/projected/c42a6620-3692-4055-8497-6826b9bb063e-kube-api-access-lv44j\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.159290 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.159081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42a6620-3692-4055-8497-6826b9bb063e-openshift-service-ca\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.159290 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.159167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-metrics-certs\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.159798 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.159773    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42a6620-3692-4055-8497-6826b9bb063e-openshift-service-ca\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.161585 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.161560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-maas-api-tls\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.161672 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.161583    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42a6620-3692-4055-8497-6826b9bb063e-metrics-certs\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.169802 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.169784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv44j\" (UniqueName: \"kubernetes.io/projected/c42a6620-3692-4055-8497-6826b9bb063e-kube-api-access-lv44j\") pod \"maas-api-8bdb6f45c-cs9js\" (UID: \"c42a6620-3692-4055-8497-6826b9bb063e\") " pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.273327 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.273238    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:51.546006 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:51.545981    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"]
Sep 18 07:08:51.548069 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:08:51.548042    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42a6620_3692_4055_8497_6826b9bb063e.slice/crio-3052908c5ed478012519ab56530cd78c8af4f10560c1514659060c9089340b77 WatchSource:0}: Error finding container 3052908c5ed478012519ab56530cd78c8af4f10560c1514659060c9089340b77: Status 404 returned error can't find the container with id 3052908c5ed478012519ab56530cd78c8af4f10560c1514659060c9089340b77
Sep 18 07:08:52.041527 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:52.041499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js" event={"ID":"c42a6620-3692-4055-8497-6826b9bb063e","Type":"ContainerStarted","Data":"3052908c5ed478012519ab56530cd78c8af4f10560c1514659060c9089340b77"}
Sep 18 07:08:53.047457 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:53.047418    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js" event={"ID":"c42a6620-3692-4055-8497-6826b9bb063e","Type":"ContainerStarted","Data":"90fb8795b3192dc132895a42bdf905a8a1b8d553b918695e9f6546951ee13e24"}
Sep 18 07:08:53.047834 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:53.047483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:53.067031 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:53.066987    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js" podStartSLOduration=2.600585124 podStartE2EDuration="3.066970074s" totalImagesPullingTime="466.38495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:08:53.065815872 +0000 UTC m=+2812.649201528" watchObservedRunningTime="2026-09-18 07:08:53.066970074 +0000 UTC m=+2812.650355726"
Sep 18 07:08:59.057704 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:59.057670    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8bdb6f45c-cs9js"
Sep 18 07:08:59.140780 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:59.140744    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 07:08:59.141007 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:08:59.140979    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerName="maas-api" containerID="cri-o://ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78" gracePeriod=30
Sep 18 07:09:00.208977 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.208932    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.30:8443/health\": dial tcp 10.132.0.30:8443: connect: connection refused"
Sep 18 07:09:00.388419 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.388390    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 07:09:00.546666 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.546561    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls\") pod \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") "
Sep 18 07:09:00.546666 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.546616    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4gln\" (UniqueName: \"kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln\") pod \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") "
Sep 18 07:09:00.546666 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.546633    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") pod \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") "
Sep 18 07:09:00.546966 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.546747    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca\") pod \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\" (UID: \"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8\") "
Sep 18 07:09:00.547180 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.547122    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" (UID: "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 07:09:00.548757 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.548727    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" (UID: "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:09:00.548863 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.548782    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" (UID: "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 07:09:00.548863 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.548791    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln" (OuterVolumeSpecName: "kube-api-access-w4gln") pod "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" (UID: "2a2e3b52-2a18-44e4-ad13-82c2e0d525e8"). InnerVolumeSpecName "kube-api-access-w4gln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 07:09:00.647550 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.647496    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:09:00.647550 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.647545    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:09:00.647550 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.647555    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4gln\" (UniqueName: \"kubernetes.io/projected/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-kube-api-access-w4gln\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:09:00.647550 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:00.647565    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\""
Sep 18 07:09:01.085496 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.085467    2567 generic.go:356] "Generic (PLEG): container finished" podID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerID="ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78" exitCode=0
Sep 18 07:09:01.085679 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.085525    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"
Sep 18 07:09:01.085679 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.085560    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" event={"ID":"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8","Type":"ContainerDied","Data":"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"}
Sep 18 07:09:01.085679 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.085617    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9" event={"ID":"2a2e3b52-2a18-44e4-ad13-82c2e0d525e8","Type":"ContainerDied","Data":"9c5036183827387378c224bc0514352da6b1537d8011b8f51fb39edd90130529"}
Sep 18 07:09:01.085679 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.085645    2567 scope.go:117] "RemoveContainer" containerID="ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"
Sep 18 07:09:01.094637 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.094617    2567 scope.go:117] "RemoveContainer" containerID="ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"
Sep 18 07:09:01.094925 ip-10-0-143-147 kubenswrapper[2567]: E0918 07:09:01.094903    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78\": container with ID starting with ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78 not found: ID does not exist" containerID="ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"
Sep 18 07:09:01.094974 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.094934    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78"} err="failed to get container status \"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78\": rpc error: code = NotFound desc = could not find container \"ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78\": container with ID starting with ec3502caae76e5660844dee6b0788dd2de054d90017a64d2392fc6e9b2a8ea78 not found: ID does not exist"
Sep 18 07:09:01.107120 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.107095    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 07:09:01.110453 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:01.110418    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc699bbb9-fbxh9"]
Sep 18 07:09:02.908736 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:02.908703    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" path="/var/lib/kubelet/pods/2a2e3b52-2a18-44e4-ad13-82c2e0d525e8/volumes"
Sep 18 07:09:19.152690 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:19.152656    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b64bb474f-h7xxg_922690f5-a238-4ad3-84fc-660fcab998e6/authorino/0.log"
Sep 18 07:09:25.124551 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:25.124526    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c8b64bc5-lsx4f_cd896cb4-8a74-4b33-9f6f-0d8b90de6700/manager/0.log"
Sep 18 07:09:25.510101 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:25.510067    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-4q4d8_1bbc1505-2348-44a4-a62c-1774b61f3136/copy-manifests/0.log"
Sep 18 07:09:25.520745 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:25.520716    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-4q4d8_1bbc1505-2348-44a4-a62c-1774b61f3136/manager/0.log"
Sep 18 07:09:26.131197 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:26.131163    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8bdb6f45c-cs9js_c42a6620-3692-4055-8497-6826b9bb063e/maas-api/0.log"
Sep 18 07:09:26.620201 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:26.620170    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d5pdc_1909799c-66c4-4efd-bc3c-8adf9d0b78b0/postgres/0.log"
Sep 18 07:09:27.446195 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:27.446164    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b64bb474f-h7xxg_922690f5-a238-4ad3-84fc-660fcab998e6/authorino/0.log"
Sep 18 07:09:27.700684 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:27.700603    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-xvrjd_0540e07a-d58f-442e-881a-051f9726ff3c/manager/0.log"
Sep 18 07:09:27.827094 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:27.827067    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d89v9_03070c95-7875-43df-96b5-48df54107303/kuadrant-console-plugin/0.log"
Sep 18 07:09:28.956434 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:28.956408    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd68dc59-wktc5_802501dc-0496-464c-bebf-9d640a6dc13f/kube-auth-proxy/0.log"
Sep 18 07:09:29.205749 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:29.205716    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4s9v7_d7b972cc-f0d8-4691-956c-89dd94200ae8/payload-pre-processing/0.log"
Sep 18 07:09:29.967183 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:29.967127    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jxczh_c97cf146-3055-4b25-a420-56320347d03c/main/0.log"
Sep 18 07:09:30.105418 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:30.105393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-dzmk7_7a966e4b-de55-4d0c-9b54-7a49613f5341/main/0.log"
Sep 18 07:09:36.213187 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.213125    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87c5n/must-gather-wxpm5"]
Sep 18 07:09:36.213762 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.213716    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerName="maas-api"
Sep 18 07:09:36.213762 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.213738    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerName="maas-api"
Sep 18 07:09:36.213898 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.213823    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2e3b52-2a18-44e4-ad13-82c2e0d525e8" containerName="maas-api"
Sep 18 07:09:36.217360 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.217337    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.221863 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.221835    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87c5n\"/\"openshift-service-ca.crt\""
Sep 18 07:09:36.221978 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.221966    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87c5n\"/\"kube-root-ca.crt\""
Sep 18 07:09:36.224493 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.224471    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87c5n\"/\"default-dockercfg-h95g6\""
Sep 18 07:09:36.246865 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.246844    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/must-gather-wxpm5"]
Sep 18 07:09:36.258529 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.258504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-must-gather-output\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.258637 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.258536    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9vm\" (UniqueName: \"kubernetes.io/projected/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-kube-api-access-5k9vm\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.359798 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.359761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-must-gather-output\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.359976 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.359809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9vm\" (UniqueName: \"kubernetes.io/projected/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-kube-api-access-5k9vm\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.360167 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.360125    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-must-gather-output\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.377253 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.377221    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9vm\" (UniqueName: \"kubernetes.io/projected/1f4b58a5-dc7f-41ef-a642-27c53a1e1a78-kube-api-access-5k9vm\") pod \"must-gather-wxpm5\" (UID: \"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78\") " pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.526916 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.526835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/must-gather-wxpm5"
Sep 18 07:09:36.831785 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.831756    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/must-gather-wxpm5"]
Sep 18 07:09:36.834286 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:09:36.834257    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f4b58a5_dc7f_41ef_a642_27c53a1e1a78.slice/crio-11d7af978b8bc1d001f55c37d637b226e41be71840c2267df0b92869c8f01c3a WatchSource:0}: Error finding container 11d7af978b8bc1d001f55c37d637b226e41be71840c2267df0b92869c8f01c3a: Status 404 returned error can't find the container with id 11d7af978b8bc1d001f55c37d637b226e41be71840c2267df0b92869c8f01c3a
Sep 18 07:09:36.836221 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:36.836200    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 07:09:37.252098 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:37.252060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/must-gather-wxpm5" event={"ID":"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78","Type":"ContainerStarted","Data":"11d7af978b8bc1d001f55c37d637b226e41be71840c2267df0b92869c8f01c3a"}
Sep 18 07:09:38.258743 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:38.258706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/must-gather-wxpm5" event={"ID":"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78","Type":"ContainerStarted","Data":"26c71e102f0ee091948899c83644629f27aee9a713361bd80c1a76b0bfdb42fc"}
Sep 18 07:09:38.259114 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:38.258755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/must-gather-wxpm5" event={"ID":"1f4b58a5-dc7f-41ef-a642-27c53a1e1a78","Type":"ContainerStarted","Data":"73c87ea07b0ad1465f82ccda2f2b8583629a7e81241cd1b285a9a653aa3ff51e"}
Sep 18 07:09:38.296944 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:38.296881    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87c5n/must-gather-wxpm5" podStartSLOduration=1.506430065 podStartE2EDuration="2.296858683s" totalImagesPullingTime="790.428618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 07:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:09:38.293957941 +0000 UTC m=+2857.877343597" watchObservedRunningTime="2026-09-18 07:09:38.296858683 +0000 UTC m=+2857.880244336"
Sep 18 07:09:39.798940 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:39.798907    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k6qw9_415bbda1-0b2a-45c3-8ae1-6d10f8e7da8f/global-pull-secret-syncer/0.log"
Sep 18 07:09:40.143823 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:40.143710    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbqkl_0959cf8e-4192-40bb-9351-5753f6d0e440/konnectivity-agent/0.log"
Sep 18 07:09:40.322354 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:40.322320    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-147.ec2.internal_65769deb7fc6587d12564f9ff3479c19/haproxy/0.log"
Sep 18 07:09:43.720744 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:43.720709    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b64bb474f-h7xxg_922690f5-a238-4ad3-84fc-660fcab998e6/authorino/0.log"
Sep 18 07:09:43.825577 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:43.825539    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-xvrjd_0540e07a-d58f-442e-881a-051f9726ff3c/manager/0.log"
Sep 18 07:09:43.859125 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:43.859097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d89v9_03070c95-7875-43df-96b5-48df54107303/kuadrant-console-plugin/0.log"
Sep 18 07:09:46.243487 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.243452    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzkr6_d5f06a2a-b1fd-4229-ba17-7416e1894094/node-exporter/0.log"
Sep 18 07:09:46.292095 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.292063    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzkr6_d5f06a2a-b1fd-4229-ba17-7416e1894094/kube-rbac-proxy/0.log"
Sep 18 07:09:46.327682 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.327656    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lzkr6_d5f06a2a-b1fd-4229-ba17-7416e1894094/init-textfile/0.log"
Sep 18 07:09:46.682859 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.682822    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/prometheus/0.log"
Sep 18 07:09:46.707994 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.707954    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/config-reloader/0.log"
Sep 18 07:09:46.743155 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.743112    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/thanos-sidecar/0.log"
Sep 18 07:09:46.797123 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.797095    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/kube-rbac-proxy-web/0.log"
Sep 18 07:09:46.841805 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.841779    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/kube-rbac-proxy/0.log"
Sep 18 07:09:46.870114 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.869906    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/kube-rbac-proxy-thanos/0.log"
Sep 18 07:09:46.928812 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.928774    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f34f2fd7-44aa-483c-a593-4cc50b728df3/init-config-reloader/0.log"
Sep 18 07:09:46.992733 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:46.992688    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-d594v_3a8dbbed-3a71-404f-a4ad-2392eff47de3/prometheus-operator/0.log"
Sep 18 07:09:47.026000 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:47.025951    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-d594v_3a8dbbed-3a71-404f-a4ad-2392eff47de3/kube-rbac-proxy/0.log"
Sep 18 07:09:47.103811 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:47.103691    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-svwhs_2a467d06-ba49-4234-aa8d-5bf7ea956240/prometheus-operator-admission-webhook/0.log"
Sep 18 07:09:49.008008 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.007968    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-gctpq_8c8c6630-d359-48a6-b8c4-057493ecd834/networking-console-plugin/0.log"
Sep 18 07:09:49.653647 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.653610    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"]
Sep 18 07:09:49.661666 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.661631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.690540 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.690504    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"]
Sep 18 07:09:49.778608 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.778488    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/2.log"
Sep 18 07:09:49.784087 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.784028    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-kggmd_4327d4f3-0131-4777-b6d5-df915eab4b4e/console-operator/3.log"
Sep 18 07:09:49.833889 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.833854    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-lib-modules\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.834081 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.834005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-proc\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.834081 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.834064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlb7v\" (UniqueName: \"kubernetes.io/projected/60745ec0-e63d-4caf-a514-6d89a515c34d-kube-api-access-zlb7v\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.834234 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.834123    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-sys\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.834234 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.834177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-podres\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.934836 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlb7v\" (UniqueName: \"kubernetes.io/projected/60745ec0-e63d-4caf-a514-6d89a515c34d-kube-api-access-zlb7v\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.934836 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-sys\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-podres\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-lib-modules\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-proc\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934969    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-sys\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.934999    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-proc\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935069 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.935019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-podres\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.935316 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.935089    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/60745ec0-e63d-4caf-a514-6d89a515c34d-lib-modules\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.945466 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.945431    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlb7v\" (UniqueName: \"kubernetes.io/projected/60745ec0-e63d-4caf-a514-6d89a515c34d-kube-api-access-zlb7v\") pod \"perf-node-gather-daemonset-bdfc8\" (UID: \"60745ec0-e63d-4caf-a514-6d89a515c34d\") " pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:49.972706 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:49.972675    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:50.258239 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:50.258203    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"]
Sep 18 07:09:50.265491 ip-10-0-143-147 kubenswrapper[2567]: W0918 07:09:50.265439    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60745ec0_e63d_4caf_a514_6d89a515c34d.slice/crio-3eaecc600b49eebc5f428d45dda3f20e3761d99f5a70db2fde6c572c1ffd0745 WatchSource:0}: Error finding container 3eaecc600b49eebc5f428d45dda3f20e3761d99f5a70db2fde6c572c1ffd0745: Status 404 returned error can't find the container with id 3eaecc600b49eebc5f428d45dda3f20e3761d99f5a70db2fde6c572c1ffd0745
Sep 18 07:09:50.343566 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:50.343473    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8" event={"ID":"60745ec0-e63d-4caf-a514-6d89a515c34d","Type":"ContainerStarted","Data":"3eaecc600b49eebc5f428d45dda3f20e3761d99f5a70db2fde6c572c1ffd0745"}
Sep 18 07:09:51.352800 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:51.352760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8" event={"ID":"60745ec0-e63d-4caf-a514-6d89a515c34d","Type":"ContainerStarted","Data":"ff44cb36621e65483f7a14747ec3e7c7ba70a74d93e86549357fafc0c7db5893"}
Sep 18 07:09:51.353394 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:51.353076    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:09:51.390164 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:51.390097    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8" podStartSLOduration=2.39007695 podStartE2EDuration="2.39007695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 07:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 07:09:51.383831298 +0000 UTC m=+2870.967216948" watchObservedRunningTime="2026-09-18 07:09:51.39007695 +0000 UTC m=+2870.973462650"
Sep 18 07:09:52.595931 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:52.595904    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cjhw_29bfe3a4-5b99-4e12-8574-e2bc18699d64/dns/0.log"
Sep 18 07:09:52.644687 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:52.644661    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cjhw_29bfe3a4-5b99-4e12-8574-e2bc18699d64/kube-rbac-proxy/0.log"
Sep 18 07:09:52.908647 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:52.908564    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lflsh_6d5ac4a2-e29f-4c7f-b9db-d8c021a09487/dns-node-resolver/0.log"
Sep 18 07:09:53.739984 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:53.739908    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fqclk_f7f1513d-e978-4051-9a08-56ad33a5e160/node-ca/0.log"
Sep 18 07:09:55.369689 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:55.369655    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd68dc59-wktc5_802501dc-0496-464c-bebf-9d640a6dc13f/kube-auth-proxy/0.log"
Sep 18 07:09:55.458182 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:55.458131    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4s9v7_d7b972cc-f0d8-4691-956c-89dd94200ae8/payload-pre-processing/0.log"
Sep 18 07:09:56.528609 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:56.528582    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vhs6h_9343c376-e916-43cb-97fd-3002665a86d3/serve-healthcheck-canary/0.log"
Sep 18 07:09:57.145955 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:57.145927    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45psb_9c27008c-d284-4b5e-8575-a368c862ec86/kube-rbac-proxy/0.log"
Sep 18 07:09:57.170561 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:57.170535    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45psb_9c27008c-d284-4b5e-8575-a368c862ec86/exporter/0.log"
Sep 18 07:09:57.197320 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:57.197294    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-45psb_9c27008c-d284-4b5e-8575-a368c862ec86/extractor/0.log"
Sep 18 07:09:57.371264 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:09:57.371235    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87c5n/perf-node-gather-daemonset-bdfc8"
Sep 18 07:10:00.308497 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:00.308464    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84c8b64bc5-lsx4f_cd896cb4-8a74-4b33-9f6f-0d8b90de6700/manager/0.log"
Sep 18 07:10:00.335927 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:00.335897    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd68dc59-wktc5_802501dc-0496-464c-bebf-9d640a6dc13f/kube-auth-proxy/0.log"
Sep 18 07:10:00.415962 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:00.415887    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4s9v7_d7b972cc-f0d8-4691-956c-89dd94200ae8/payload-pre-processing/0.log"
Sep 18 07:10:00.594030 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:00.593999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-4q4d8_1bbc1505-2348-44a4-a62c-1774b61f3136/manager/0.log"
Sep 18 07:10:00.649148 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:00.649116    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fd575c64f-4q4d8_1bbc1505-2348-44a4-a62c-1774b61f3136/copy-manifests/0.log"
Sep 18 07:10:02.402503 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:02.402415    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64c86f57d4-68vm7_a5062625-380d-4384-af3a-037c8de0f9ac/manager/0.log"
Sep 18 07:10:02.501089 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:02.501058    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jmznw_2fac51fd-5f4e-4dc9-a611-93e1e74ca2d9/openshift-lws-operator/0.log"
Sep 18 07:10:02.738974 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:02.738939    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd68dc59-wktc5_802501dc-0496-464c-bebf-9d640a6dc13f/kube-auth-proxy/0.log"
Sep 18 07:10:02.798895 ip-10-0-143-147 kubenswrapper[2567]: I0918 07:10:02.798831    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4s9v7_d7b972cc-f0d8-4691-956c-89dd94200ae8/payload-pre-processing/0.log"