Sep 08 00:00:43.069779 ip-10-0-128-150 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 00:00:43.069793 ip-10-0-128-150 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 00:00:43.069803 ip-10-0-128-150 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 00:00:43.070228 ip-10-0-128-150 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 00:00:54.296359 ip-10-0-128-150 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 00:00:54.296374 ip-10-0-128-150 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 544f0bc3b32c41179fe9ad45554b2e9e --
Sep 08 00:02:47.708860 ip-10-0-128-150 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 00:02:48.127733 ip-10-0-128-150 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:02:48.127733 ip-10-0-128-150 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 00:02:48.127733 ip-10-0-128-150 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:02:48.127733 ip-10-0-128-150 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 00:02:48.127733 ip-10-0-128-150 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 00:02:48.128551 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.128447    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 00:02:48.131959 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131944    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.131959 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131959    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131963    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131967    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131971    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131974    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131977    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131979    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131982    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131985    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131988    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131990    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131993    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.131997    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132001    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132004    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132007    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132010    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132013    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132021    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132025    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.132020 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132028    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132032    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132035    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132038    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132041    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132044    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132047    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132049    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132052    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132055    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132058    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132062    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132066    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132068    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132071    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132074    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132076    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132079    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132082    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132084    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.132502 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132087    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132090    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132092    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132095    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132097    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132099    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132102    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132104    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132107    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132109    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132112    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132114    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132117    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132119    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132123    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132125    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132128    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132131    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132133    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.133054 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132136    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132139    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132141    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132144    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132147    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132151    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132154    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132158    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132161    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132163    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132166    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132170    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132173    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132176    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132178    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132181    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132184    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132186    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132189    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.133508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132191    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132194    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132196    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132199    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132201    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132204    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132206    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132208    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132212    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132215    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132218    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132220    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132223    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132226    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132228    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132231    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132234    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132236    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132239    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132241    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.133968 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132930    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132937    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132940    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132942    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132946    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132950    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132952    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132955    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132958    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132960    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132963    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132965    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132970    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132975    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132980    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132985    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132989    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132994    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.132998    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133002    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.134441 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133006    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133009    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133012    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133015    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133017    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133020    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133022    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133026    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133028    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133031    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133033    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133036    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133038    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133041    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133043    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133046    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133048    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133051    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133057    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.134958 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133060    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133063    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133066    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133069    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133072    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133074    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133077    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133080    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133082    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133085    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133087    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133090    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133092    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133098    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133101    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133104    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133106    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133109    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133112    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133114    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.135455 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133117    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133119    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133122    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133124    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133127    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133129    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133132    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133134    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133137    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133141    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133144    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133146    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133149    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133152    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133154    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133157    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133159    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133162    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133165    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133167    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.135961 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133170    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133174    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133176    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133179    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133182    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133188    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133190    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133193    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133195    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133198    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133201    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133204    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133206    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133209    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133211    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133214    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133216    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133219    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133221    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133224    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.136448 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133607    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133612    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133615    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133618    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133621    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133624    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133627    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133630    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133632    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133635    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133637    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133640    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133643    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133645    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133648    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133650    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133653    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133660    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133664    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.136957 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133667    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133670    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133673    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133676    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133679    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133682    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133685    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133687    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133690    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133692    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133695    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133697    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133700    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133703    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133707    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133710    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133713    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133716    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133719    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.137428 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133722    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133725    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133728    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133730    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133733    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133735    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133738    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133741    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133743    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133747    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133749    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133754    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133757    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133760    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133763    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133765    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133768    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133770    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133773    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133775    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.137913 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133778    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133781    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133785    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133788    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133790    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133793    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133795    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133798    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133800    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133803    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133805    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133808    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133811    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133813    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133816    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133819    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133821    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133824    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133826    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133828    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.138401 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133831    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133833    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133836    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133840    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133843    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133845    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133848    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133850    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133853    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133856    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133858    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133862    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133864    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133867    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133869    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133872    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133874    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133877    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133879    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133882    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.138892 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.133885    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.133957    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.133991    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.133996    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134001    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134006    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134009    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134013    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134017    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134021    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134023    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134027    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134030    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134033    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134036    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134039    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134042    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134045    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134048    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134053    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134055    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134058    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134062    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134065    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 00:02:48.139388 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134069    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134072    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134075    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134078    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134081    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134084    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134086    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134089    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134092    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134096    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134099    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134102    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134105    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134108    2580 flags.go:64] FLAG: --enable-server="true"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134111    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134591    2580 flags.go:64] FLAG: --event-burst="100"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134595    2580 flags.go:64] FLAG: --event-qps="50"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134598    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134602    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134605    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134609    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134612    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134614    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134619    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134623    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 00:02:48.139970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134626    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134628    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134631    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134635    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134638    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134641    2580 flags.go:64] FLAG: --feature-gates=""
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134645    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134648    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134651    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134654    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134657    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134660    2580 flags.go:64] FLAG: --help="false"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134663    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134666    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134669    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134672    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134675    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134678    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134681    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134684    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134687    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134690    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134693    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134697    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 00:02:48.140649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134700    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134702    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134705    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134708    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134711    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134714    2580 flags.go:64] FLAG: --lock-file=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134718    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134721    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134724    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134730    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134732    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134735    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134739    2580 flags.go:64] FLAG: --logging-format="text"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134742    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134745    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134748    2580 flags.go:64] FLAG: --manifest-url=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134751    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134756    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134759    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134762    2580 flags.go:64] FLAG: --max-pods="110"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134765    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134768    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134771    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134774    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134777    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 00:02:48.141315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134780    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134783    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134790    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134793    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134796    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134799    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134802    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134808    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134811    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134814    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134816    2580 flags.go:64] FLAG: --port="10250"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134819    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134822    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e01282137d381889"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134826    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134829    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134832    2580 flags.go:64] FLAG: --register-node="true"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134835    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134838    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134841    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134844    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134847    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134850    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134853    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134856    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134859    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 00:02:48.141968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134862    2580 flags.go:64] FLAG: --runonce="false"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134864    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134867    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134870    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134873    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134876    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134879    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134882    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134885    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134888    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134891    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134895    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134898    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134901    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134905    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134907    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134913    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134915    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134918    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134923    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134929    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134932    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134935    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134938    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134941    2580 flags.go:64] FLAG: --v="2"
Sep 08 00:02:48.142668 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134945    2580 flags.go:64] FLAG: --version="false"
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134949    2580 flags.go:64] FLAG: --vmodule=""
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134956    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.134959    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135079    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135083    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135086    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135089    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135091    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135094    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135097    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135099    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135102    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135104    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135107    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135111    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135114    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135118    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135122    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.143291 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135125    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135127    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135130    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135133    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135136    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135140    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135143    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135146    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135150    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135154    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135156    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135159    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135162    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135164    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135167    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135170    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135173    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135175    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135178    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.143770 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135180    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135183    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135185    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135188    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135190    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135193    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135195    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135198    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135201    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135203    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135206    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135208    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135212    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135215    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135217    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135220    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135222    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135225    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135228    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135230    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.144499 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135233    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135237    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135240    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135242    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135245    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135247    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135250    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135252    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135255    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135257    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135260    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135263    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135265    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135268    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135270    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135273    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135275    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135278    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135280    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135283    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.145080 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135285    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135288    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135291    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135293    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135297    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135299    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135302    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135304    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135307    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135309    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135312    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135315    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135317    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135321    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135324    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135326    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135336    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135338    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135341    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135343    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.145643 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135346    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135350    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135353    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.135359    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.135800    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.142790    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.142805    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.142821    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.142827    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142885    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142891    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142894    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142897    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.146152 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142899    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142902    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142905    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142908    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142911    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142916    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142919    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142922    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142925    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142927    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142930    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142932    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142935    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142938    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142940    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142943    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142945    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142948    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142951    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.146508 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142953    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142956    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142959    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142961    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142964    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142967    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142971    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142974    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142976    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142979    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142981    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142984    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142986    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142988    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142991    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142994    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142996    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.142999    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143003    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.146998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143007    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143010    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143012    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143015    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143019    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143022    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143025    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143028    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143031    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143033    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143037    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143039    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143042    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143044    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143047    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143050    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143052    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143055    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143058    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143061    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.147475 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143064    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143074    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143077    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143080    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143082    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143085    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143088    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143090    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143093    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143095    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143098    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143100    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143103    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143105    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143108    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143111    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143113    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143116    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143118    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143121    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.147982 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143123    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143125    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143128    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143131    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143133    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143136    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143139    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143141    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143144    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143146    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143149    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143152    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143154    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143157    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143162    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.148515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143165    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.143170    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143302    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143306    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143309    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143312    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143315    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143318    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143321    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143324    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143327    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143329    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143332    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143335    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143338    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 00:02:48.148970 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143340    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143343    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143346    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143349    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143352    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143354    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143357    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143360    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143362    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143365    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143367    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143370    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143372    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143375    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143377    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143380    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143383    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143386    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143388    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143391    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 00:02:48.149392 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143393    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143396    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143398    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143402    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143406    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143408    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143411    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143413    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143416    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143418    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143421    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143423    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143426    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143428    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143431    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143433    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143436    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143439    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143442    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 00:02:48.150037 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143445    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143448    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143450    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143453    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143457    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143460    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143463    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143465    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143468    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143471    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143474    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143476    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143478    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143481    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143484    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143486    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143489    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143491    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143493    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 00:02:48.150515 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143496    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143498    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143501    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143503    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143506    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143508    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143510    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143513    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143515    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143518    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143521    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143523    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143538    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143541    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143543    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143546    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143548    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143551    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143553    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143556    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 00:02:48.150998 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143559    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143561    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143563    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143566    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143569    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143571    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143574    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:48.143576    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.143581    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.145427    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.148434    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.149260    2580 server.go:1019] "Starting client certificate rotation"
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.149363    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 00:02:48.151489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.149413    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 00:02:48.151852 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.151463    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 00:02:48.152551 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.152534    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 00:02:48.162401 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.162382    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 08 00:02:48.166589 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.166572    2580 log.go:25] "Validated CRI v1 image API"
Sep 08 00:02:48.168076 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.168060    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 00:02:48.170845 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.170826    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cc4d9ee1-d3d5-473d-8254-ef8d5d049dc5:/dev/nvme0n1p4 e88f5d74-721c-4042-b1b8-6571b90760fc:/dev/nvme0n1p3]
Sep 08 00:02:48.170889 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.170845    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 00:02:48.177836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.177437    2580 manager.go:216] Machine: {Timestamp:2026-09-08 00:02:48.175937145 +0000 UTC m=+0.348202020 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108296 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29aafaaa0521ef642c01db42efd35d SystemUUID:ec29aafa-aa05-21ef-642c-01db42efd35d BootID:544f0bc3-b32c-4117-9fe9-ad45554b2e9e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:58:9c:7f:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:58:9c:7f:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:00:77:b1:59:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 00:02:48.177836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.177826    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 00:02:48.177952 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.177915    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 00:02:48.178503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.178487    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 00:02:48.180842 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.180816    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 00:02:48.180972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.180845    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-150.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 00:02:48.181511 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.181501    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 00:02:48.181551 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.181514    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 00:02:48.181551 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.181521    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 00:02:48.182227 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.182217    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 00:02:48.183482 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.183471    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 00:02:48.186606 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186595    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 00:02:48.186712 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186703    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 00:02:48.186749 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186732    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 00:02:48.186749 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186743    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 00:02:48.186811 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186755    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 00:02:48.186811 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186765    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 08 00:02:48.186811 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.186773    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 00:02:48.187106 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.187093    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 00:02:48.187152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.187118    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 00:02:48.188915 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.188896    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 00:02:48.191343 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191330    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 00:02:48.191389 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191360    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 00:02:48.191516 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191506    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 00:02:48.191561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191522    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 00:02:48.191561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191543    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 00:02:48.191561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191553    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 00:02:48.191645 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191564    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 00:02:48.191645 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191571    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 00:02:48.191981 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191971    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 00:02:48.192011 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191983    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 00:02:48.192011 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191991    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 00:02:48.192011 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.191996    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 00:02:48.192011 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.192005    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 00:02:48.192011 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.192013    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 00:02:48.192576 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.192565    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 00:02:48.192613 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.192578    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 00:02:48.196145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.196095    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-150.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 00:02:48.196145 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.196124    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 00:02:48.196276 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.196182    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 00:02:48.196581 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.196564    2580 server.go:1268] "Started kubelet"
Sep 08 00:02:48.196666 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.196626    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 00:02:48.196760 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.196709    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 00:02:48.196812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.196779    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 00:02:48.197387 ip-10-0-128-150 systemd[1]: Started Kubernetes Kubelet.
Sep 08 00:02:48.197759 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.197739    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-82nrx"
Sep 08 00:02:48.198189 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.198023    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 00:02:48.199390 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.199374    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 00:02:48.200922 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.200904    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 00:02:48.202643 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.202625    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 00:02:48.202643 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.202639    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 00:02:48.202869 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.202848    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.203245 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203231    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 00:02:48.203353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203332    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 00:02:48.203404 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203354    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 00:02:48.203480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203455    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 00:02:48.203480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203475    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 00:02:48.203928 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203903    2580 factory.go:153] Registering CRI-O factory
Sep 08 00:02:48.203928 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.203941    2580 factory.go:223] Registration of the crio container factory successfully
Sep 08 00:02:48.204077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204008    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 00:02:48.204077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204018    2580 factory.go:55] Registering systemd factory
Sep 08 00:02:48.204077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204026    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 08 00:02:48.204077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204042    2580 factory.go:103] Registering Raw factory
Sep 08 00:02:48.204077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204055    2580 manager.go:1196] Started watching for new ooms in manager
Sep 08 00:02:48.204478 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.203280    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-150.ec2.internal.18d32f8efe6b3d35  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-150.ec2.internal,UID:ip-10-0-128-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-150.ec2.internal,},FirstTimestamp:2026-09-08 00:02:48.196521269 +0000 UTC m=+0.368786144,LastTimestamp:2026-09-08 00:02:48.196521269 +0000 UTC m=+0.368786144,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-150.ec2.internal,}"
Sep 08 00:02:48.204602 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204563    2580 manager.go:349] Starting recovery of all containers
Sep 08 00:02:48.204766 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.204742    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-82nrx"
Sep 08 00:02:48.212205 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.212180    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:48.215989 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.215962    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-150.ec2.internal\" not found" node="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.218312 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.218293    2580 manager.go:354] Recovery completed
Sep 08 00:02:48.223093 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.223059    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.224993 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.224980    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.225051 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225005    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.225051 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225016    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.225731 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225588    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 00:02:48.225731 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225599    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 00:02:48.225731 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225621    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 00:02:48.225731 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.225603    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.226602 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226584    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.226685 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226614    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.226685 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226628    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.226791 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226739    2580 policy_none.go:49] "None policy: Start"
Sep 08 00:02:48.226791 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226753    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 00:02:48.226791 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.226763    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 00:02:48.227865 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.227853    2580 policy_none.go:47] "Start"
Sep 08 00:02:48.278185 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278170    2580 manager.go:337] "Starting Device Plugin manager"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.278221    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278235    2580 server.go:87] "Starting device plugin registration server"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278469    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278479    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278590    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278659    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.278667    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.279164    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 00:02:48.284597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.279214    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.322761 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.322729    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 00:02:48.323890 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.323865    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 00:02:48.323975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.323924    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 00:02:48.323975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.323953    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 00:02:48.324068 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.323998    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 00:02:48.325998 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.325980    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:48.379115 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.379027    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.380441 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.380423    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.380561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.380455    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.380561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.380467    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.380561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.380488    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.387631 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.387616    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.387674 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.387639    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-150.ec2.internal\": node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.400933 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.400913    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.424459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.424430    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"]
Sep 08 00:02:48.424518 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.424498    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.425898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.425875    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.426008 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.425903    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.426008 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.425914    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.427195 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.427183    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.427328 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.427314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.427386 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.427348    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.429857 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429834    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.429857 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429845    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.429857 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429859    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.429857 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429863    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.430093 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429873    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.430093 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.429872    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.430975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.430959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.431050 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.430990    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 00:02:48.432333 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.432318    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 00:02:48.432421 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.432341    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 00:02:48.432421 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.432350    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeHasSufficientPID"
Sep 08 00:02:48.461183 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.461149    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-150.ec2.internal\" not found" node="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.465562 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.465544    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-150.ec2.internal\" not found" node="ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.501437 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.501418    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.505661 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.505646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b2683d5f56405395b041d9bac31f181-config\") pod \"kube-apiserver-proxy-ip-10-0-128-150.ec2.internal\" (UID: \"1b2683d5f56405395b041d9bac31f181\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.505721 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.505670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.505721 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.505687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.602459 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.602407    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.606697 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.606764 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.606764 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b2683d5f56405395b041d9bac31f181-config\") pod \"kube-apiserver-proxy-ip-10-0-128-150.ec2.internal\" (UID: \"1b2683d5f56405395b041d9bac31f181\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.606764 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b2683d5f56405395b041d9bac31f181-config\") pod \"kube-apiserver-proxy-ip-10-0-128-150.ec2.internal\" (UID: \"1b2683d5f56405395b041d9bac31f181\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.606861 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.606861 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.606782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b43182a60cdc7752da5f71487404314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal\" (UID: \"6b43182a60cdc7752da5f71487404314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.702526 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.702498    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.764313 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.764278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.769049 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:48.768801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:48.803488 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.803450    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:48.908809 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:48.908752    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.009853 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.009751    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.084523 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.084491    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:49.110251 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.110204    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.151513 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.151483    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 00:02:49.151999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.151651    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 00:02:49.151999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.151676    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 00:02:49.151999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.151674    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 00:02:49.207274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.207231    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 23:57:48 +0000 UTC" deadline="2028-05-06 06:50:42.190120252 +0000 UTC"
Sep 08 00:02:49.207274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.207262    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14550h47m52.982861331s"
Sep 08 00:02:49.210985 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.210967    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.311773 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.311736    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.391860 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:49.391816    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b43182a60cdc7752da5f71487404314.slice/crio-228472d5afeccb7a70857a2382e4db1c965bf762f567b118cfc78205c8795c21 WatchSource:0}: Error finding container 228472d5afeccb7a70857a2382e4db1c965bf762f567b118cfc78205c8795c21: Status 404 returned error can't find the container with id 228472d5afeccb7a70857a2382e4db1c965bf762f567b118cfc78205c8795c21
Sep 08 00:02:49.392341 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:49.392316    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2683d5f56405395b041d9bac31f181.slice/crio-b07ba2810d7b7d21c02aa7c44295d827174b791607e0a07efcf9cad6b4697dc8 WatchSource:0}: Error finding container b07ba2810d7b7d21c02aa7c44295d827174b791607e0a07efcf9cad6b4697dc8: Status 404 returned error can't find the container with id b07ba2810d7b7d21c02aa7c44295d827174b791607e0a07efcf9cad6b4697dc8
Sep 08 00:02:49.396514 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.396498    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:02:49.412706 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.412680    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.513117 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.513077    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.614020 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.613932    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.714134 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:49.714104    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-150.ec2.internal\" not found"
Sep 08 00:02:49.723372 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.723351    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:49.803227 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.803189    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:49.820808 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.820783    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 00:02:49.821825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.821810    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal"
Sep 08 00:02:49.829112 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:49.829096    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 00:02:50.190017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.189984    2580 apiserver.go:52] "Watching apiserver"
Sep 08 00:02:50.199013 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.198693    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 00:02:50.199268 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.199247    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-j9n55","openshift-multus/network-metrics-daemon-54s8c","openshift-network-operator/iptables-alerter-cnsbf","kube-system/global-pull-secret-syncer-xpvcr","kube-system/konnectivity-agent-qrfmj","kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal","openshift-dns/node-resolver-5f47g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal","openshift-network-diagnostics/network-check-target-rf9cf","openshift-ovn-kubernetes/ovnkube-node-llnkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2","openshift-cluster-node-tuning-operator/tuned-ztmxv","openshift-image-registry/node-ca-ppdfm","openshift-multus/multus-additional-cni-plugins-fc9f2"]
Sep 08 00:02:50.200811 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.200787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.202040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.202020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.203056 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.203035    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.203181 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.203041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dfkcx\""
Sep 08 00:02:50.203495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.203447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.203954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.203933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.204133 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.204098    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 00:02:50.204760 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.204737    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 00:02:50.204886 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.204788    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.204886 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.204872    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.205043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.204968    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-klw5t\""
Sep 08 00:02:50.205043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.205037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.205187 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.205105    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:02:50.206059 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.205597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 00:02:50.206059 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.205905    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.206059 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.205974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.206059 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.206024    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r87wj\""
Sep 08 00:02:50.206386 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.206234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.207413 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.207393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.209327 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.208984    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76fwz\""
Sep 08 00:02:50.209327 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.209031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 00:02:50.209327 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.209237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 00:02:50.209871 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.209850    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 00:02:50.210220 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210201    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 00:02:50.210339 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210269    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.210442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210424    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.210510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210494    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 00:02:50.210622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w4hw9\""
Sep 08 00:02:50.210781 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.210760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:50.210886 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.210832    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:02:50.211974 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.211956    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.213353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.213330    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.213452 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.213405    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:02:50.214065 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.214038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pgtq5\""
Sep 08 00:02:50.214635 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.214300    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 00:02:50.215026 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.214950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.215391 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.215367    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 00:02:50.216476 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.216460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.217161 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f1e5563-87ef-49d6-b328-2b208ea3da80-serviceca\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.217256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdnv\" (UniqueName: \"kubernetes.io/projected/1c37a23d-2a77-4773-af02-1ea914005127-kube-api-access-sfdnv\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.217256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-dbus\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.217256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6r5q\" (UniqueName: \"kubernetes.io/projected/7f1e5563-87ef-49d6-b328-2b208ea3da80-kube-api-access-n6r5q\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.217405 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c37a23d-2a77-4773-af02-1ea914005127-host-slash\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.217405 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-hosts-file\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.217715 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-tmp-dir\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.217812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f1e5563-87ef-49d6-b328-2b208ea3da80-host\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217882    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-kubelet-config\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217898    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qggnf\""
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.217983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2nf\" (UniqueName: \"kubernetes.io/projected/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-kube-api-access-nt2nf\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.218022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c37a23d-2a77-4773-af02-1ea914005127-iptables-alerter-script\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.218549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.218367    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.218977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.218880    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.219465 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.219443    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 00:02:50.219547 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.219465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 00:02:50.219547 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.219521    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.220291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.219833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 00:02:50.220291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.219858    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b8qc7\""
Sep 08 00:02:50.220291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.220110    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.220291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.220168    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 00:02:50.221485 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.221464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.223678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.223659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xf544\""
Sep 08 00:02:50.224806 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.224518    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 00:02:50.225189 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.224560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:02:50.238489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.238466    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5b7rl"
Sep 08 00:02:50.246174 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.246105    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5b7rl"
Sep 08 00:02:50.304404 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.304379    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 00:02:50.318621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzgc\" (UniqueName: \"kubernetes.io/projected/4ed3c48f-fe87-489a-b011-76cb7b05fe30-kube-api-access-flzgc\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.318744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-systemd-units\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.318744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-ovn\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.318744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-node-log\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.318744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-conf-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-lib-modules\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-host\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-kubelet-config\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-systemd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-config\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-kubelet-config\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-cni-binary-copy\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.318968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.318966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-daemon-config\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-etc-kubernetes\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f1e5563-87ef-49d6-b328-2b208ea3da80-serviceca\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdnv\" (UniqueName: \"kubernetes.io/projected/1c37a23d-2a77-4773-af02-1ea914005127-kube-api-access-sfdnv\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-etc-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-log-socket\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-tmp\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-kubelet\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-modprobe-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-run\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-slash\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.319466 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-script-lib\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-system-cni-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2nf\" (UniqueName: \"kubernetes.io/projected/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-kube-api-access-nt2nf\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-bin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5tv\" (UniqueName: \"kubernetes.io/projected/583e7bd9-a4cb-4a45-9a66-255aef584166-kube-api-access-jd5tv\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszjd\" (UniqueName: \"kubernetes.io/projected/b84de74e-3f1f-4743-920d-4f625c76a3f0-kube-api-access-zszjd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-registration-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbrl\" (UniqueName: \"kubernetes.io/projected/10e0c506-9b9d-4525-a5e5-e30569246604-kube-api-access-kqbrl\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qcr\" (UniqueName: \"kubernetes.io/projected/2ec924f8-1839-40bd-84b0-9220783b414e-kube-api-access-j8qcr\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-var-lib-kubelet\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-tuned\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c37a23d-2a77-4773-af02-1ea914005127-host-slash\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319842    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-device-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-hostroot\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.320039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-kubernetes\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c37a23d-2a77-4773-af02-1ea914005127-host-slash\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxl68\" (UniqueName: \"kubernetes.io/projected/77a53b91-7079-40c5-a9d0-1cf368aac1ef-kube-api-access-vxl68\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cnibin\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.319969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-hosts-file\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-tmp-dir\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-hosts-file\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f1e5563-87ef-49d6-b328-2b208ea3da80-host\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f1e5563-87ef-49d6-b328-2b208ea3da80-serviceca\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-env-overrides\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f1e5563-87ef-49d6-b328-2b208ea3da80-host\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cbcd392-b02f-4a8b-b6b3-97713445f430-agent-certs\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-conf\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c37a23d-2a77-4773-af02-1ea914005127-iptables-alerter-script\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-tmp-dir\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.320695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-kubelet\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-cnibin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-multus\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysconfig\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-system-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6r5q\" (UniqueName: \"kubernetes.io/projected/7f1e5563-87ef-49d6-b328-2b208ea3da80-kube-api-access-n6r5q\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcd392-b02f-4a8b-b6b3-97713445f430-konnectivity-ca\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-binary-copy\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.320872    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-netns\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-netd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.320947    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:50.8209249 +0000 UTC m=+2.993189793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:50.321537 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-etc-selinux\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c37a23d-2a77-4773-af02-1ea914005127-iptables-alerter-script\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.320994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-sys-fs\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-multus-certs\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-sys\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-bin\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321122    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-socket-dir-parent\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-k8s-cni-cncf-io\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-netns\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-dbus\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-var-lib-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-socket-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-systemd\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.322063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.321494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b228f4d3-1a89-4fa8-87a0-78a71024d15e-dbus\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.324994 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.324966    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 00:02:50.329587 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.329428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdnv\" (UniqueName: \"kubernetes.io/projected/1c37a23d-2a77-4773-af02-1ea914005127-kube-api-access-sfdnv\") pod \"iptables-alerter-cnsbf\" (UID: \"1c37a23d-2a77-4773-af02-1ea914005127\") " pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.329587 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.329428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2nf\" (UniqueName: \"kubernetes.io/projected/c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997-kube-api-access-nt2nf\") pod \"node-resolver-5f47g\" (UID: \"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997\") " pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.330212 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.329889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6r5q\" (UniqueName: \"kubernetes.io/projected/7f1e5563-87ef-49d6-b328-2b208ea3da80-kube-api-access-n6r5q\") pod \"node-ca-ppdfm\" (UID: \"7f1e5563-87ef-49d6-b328-2b208ea3da80\") " pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.333623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.333571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal" event={"ID":"1b2683d5f56405395b041d9bac31f181","Type":"ContainerStarted","Data":"b07ba2810d7b7d21c02aa7c44295d827174b791607e0a07efcf9cad6b4697dc8"}
Sep 08 00:02:50.335571 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.335528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal" event={"ID":"6b43182a60cdc7752da5f71487404314","Type":"ContainerStarted","Data":"228472d5afeccb7a70857a2382e4db1c965bf762f567b118cfc78205c8795c21"}
Sep 08 00:02:50.406747 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.406718    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:50.422305 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-multus-certs\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422305 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-sys\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-bin\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-sys\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-multus-certs\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-bin\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-socket-dir-parent\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-k8s-cni-cncf-io\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-netns\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-socket-dir-parent\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-var-lib-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-socket-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-netns\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-systemd\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-run-k8s-cni-cncf-io\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-systemd\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-var-lib-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-socket-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzgc\" (UniqueName: \"kubernetes.io/projected/4ed3c48f-fe87-489a-b011-76cb7b05fe30-kube-api-access-flzgc\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-systemd-units\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-ovn\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-node-log\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.422960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-conf-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.422988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-lib-modules\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-host\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-systemd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-node-log\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-config\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-systemd-units\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-cni-binary-copy\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-ovn\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-daemon-config\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-host\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-etc-kubernetes\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423223    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-conf-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-etc-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-log-socket\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.423272    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:50.423821 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.423338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:50.923320319 +0000 UTC m=+3.095585211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-lib-modules\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-tmp\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-kubelet\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-modprobe-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-run\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-slash\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-script-lib\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-system-cni-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-kubelet\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-bin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5tv\" (UniqueName: \"kubernetes.io/projected/583e7bd9-a4cb-4a45-9a66-255aef584166-kube-api-access-jd5tv\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszjd\" (UniqueName: \"kubernetes.io/projected/b84de74e-3f1f-4743-920d-4f625c76a3f0-kube-api-access-zszjd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-registration-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbrl\" (UniqueName: \"kubernetes.io/projected/10e0c506-9b9d-4525-a5e5-e30569246604-kube-api-access-kqbrl\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423811    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qcr\" (UniqueName: \"kubernetes.io/projected/2ec924f8-1839-40bd-84b0-9220783b414e-kube-api-access-j8qcr\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.424812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-var-lib-kubelet\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-tuned\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-device-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-hostroot\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-kubernetes\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxl68\" (UniqueName: \"kubernetes.io/projected/77a53b91-7079-40c5-a9d0-1cf368aac1ef-kube-api-access-vxl68\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cnibin\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-env-overrides\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-config\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cbcd392-b02f-4a8b-b6b3-97713445f430-agent-certs\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-conf\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-systemd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-kubelet\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-cnibin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.425671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-cni-binary-copy\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-slash\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-multus\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-modprobe-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-run\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-multus\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysconfig\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-daemon-config\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-system-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-system-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcd392-b02f-4a8b-b6b3-97713445f430-konnectivity-ca\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysconfig\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-binary-copy\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovnkube-script-lib\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.426600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-netns\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-system-cni-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-netd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-cni-netd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-etc-selinux\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-sys-fs\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-run-netns\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-kubernetes\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-sys-fs\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-multus-cni-dir\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-host-var-lib-cni-bin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cnibin\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-registration-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b84de74e-3f1f-4743-920d-4f625c76a3f0-env-overrides\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.425968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-etc-kubernetes\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-run-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.428145 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-etc-openvswitch\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ed3c48f-fe87-489a-b011-76cb7b05fe30-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-var-lib-kubelet\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-binary-copy\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-device-dir\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-hostroot\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-log-socket\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-kubelet\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.426923    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-conf\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-sysctl-d\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.427152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/583e7bd9-a4cb-4a45-9a66-255aef584166-cnibin\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.424879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b84de74e-3f1f-4743-920d-4f625c76a3f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.423195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.427299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10e0c506-9b9d-4525-a5e5-e30569246604-etc-selinux\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.427950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b84de74e-3f1f-4743-920d-4f625c76a3f0-ovn-node-metrics-cert\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.428015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed3c48f-fe87-489a-b011-76cb7b05fe30-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.428136    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.428158    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:02:50.429047 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.428174    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:50.430009 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.428239    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:02:50.928222522 +0000 UTC m=+3.100487403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:50.430009 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.429369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-etc-tuned\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.431741 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.431691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cbcd392-b02f-4a8b-b6b3-97713445f430-konnectivity-ca\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.431848 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.431801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzgc\" (UniqueName: \"kubernetes.io/projected/4ed3c48f-fe87-489a-b011-76cb7b05fe30-kube-api-access-flzgc\") pod \"multus-additional-cni-plugins-fc9f2\" (UID: \"4ed3c48f-fe87-489a-b011-76cb7b05fe30\") " pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.432600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.432557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cbcd392-b02f-4a8b-b6b3-97713445f430-agent-certs\") pod \"konnectivity-agent-qrfmj\" (UID: \"0cbcd392-b02f-4a8b-b6b3-97713445f430\") " pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.434727 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.434703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a53b91-7079-40c5-a9d0-1cf368aac1ef-tmp\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.434832 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.434814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxl68\" (UniqueName: \"kubernetes.io/projected/77a53b91-7079-40c5-a9d0-1cf368aac1ef-kube-api-access-vxl68\") pod \"tuned-ztmxv\" (UID: \"77a53b91-7079-40c5-a9d0-1cf368aac1ef\") " pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.435361 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.435316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbrl\" (UniqueName: \"kubernetes.io/projected/10e0c506-9b9d-4525-a5e5-e30569246604-kube-api-access-kqbrl\") pod \"aws-ebs-csi-driver-node-gj2f2\" (UID: \"10e0c506-9b9d-4525-a5e5-e30569246604\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.436166 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.436113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszjd\" (UniqueName: \"kubernetes.io/projected/b84de74e-3f1f-4743-920d-4f625c76a3f0-kube-api-access-zszjd\") pod \"ovnkube-node-llnkn\" (UID: \"b84de74e-3f1f-4743-920d-4f625c76a3f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.436713 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.436685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qcr\" (UniqueName: \"kubernetes.io/projected/2ec924f8-1839-40bd-84b0-9220783b414e-kube-api-access-j8qcr\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.437424 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.437401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5tv\" (UniqueName: \"kubernetes.io/projected/583e7bd9-a4cb-4a45-9a66-255aef584166-kube-api-access-jd5tv\") pod \"multus-j9n55\" (UID: \"583e7bd9-a4cb-4a45-9a66-255aef584166\") " pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.515660 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.515594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5f47g"
Sep 08 00:02:50.524779 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.524758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ppdfm"
Sep 08 00:02:50.526170 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.526145    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:50.535921 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.535893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:02:50.542575 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.542556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnsbf"
Sep 08 00:02:50.550244 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.550226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:02:50.558899 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.558877    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fc9f2"
Sep 08 00:02:50.572567 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.572542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j9n55"
Sep 08 00:02:50.580404 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.580382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2"
Sep 08 00:02:50.586314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.586294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztmxv"
Sep 08 00:02:50.828136 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.828041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:50.828313 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.828237    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:50.828385 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.828317    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:51.828295331 +0000 UTC m=+4.000560215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:50.929131 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.929087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:50.929326 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:50.929148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:50.929326 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929298    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:50.929431 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929393    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:51.929368812 +0000 UTC m=+4.101633693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:50.929431 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929298    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:02:50.929431 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929430    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:02:50.929629 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929455    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:50.929629 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:50.929515    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:02:51.929496447 +0000 UTC m=+4.101761314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:51.103829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.103748    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 00:02:51.248318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.248283    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 23:57:50 +0000 UTC" deadline="2028-03-29 12:40:02.360666903 +0000 UTC"
Sep 08 00:02:51.248318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.248312    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13644h37m11.112358619s"
Sep 08 00:02:51.325113 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.325069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:51.325281 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.325239    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:02:51.436026 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.435916    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583e7bd9_a4cb_4a45_9a66_255aef584166.slice/crio-d7a3cac751c93eeb8e055a43cd41dea4ba3853fd83599e82315dcd37c5ab371f WatchSource:0}: Error finding container d7a3cac751c93eeb8e055a43cd41dea4ba3853fd83599e82315dcd37c5ab371f: Status 404 returned error can't find the container with id d7a3cac751c93eeb8e055a43cd41dea4ba3853fd83599e82315dcd37c5ab371f
Sep 08 00:02:51.438879 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.438723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84de74e_3f1f_4743_920d_4f625c76a3f0.slice/crio-3a35f15de2d416277d2750840c0e2994ddb33f2f01d674a1d25bb3594322fda4 WatchSource:0}: Error finding container 3a35f15de2d416277d2750840c0e2994ddb33f2f01d674a1d25bb3594322fda4: Status 404 returned error can't find the container with id 3a35f15de2d416277d2750840c0e2994ddb33f2f01d674a1d25bb3594322fda4
Sep 08 00:02:51.444997 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.444810    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a53b91_7079_40c5_a9d0_1cf368aac1ef.slice/crio-a4463385ab6417b760474bca1503b3a0c7ceafb6d5c11989b37ec33ee4a866e8 WatchSource:0}: Error finding container a4463385ab6417b760474bca1503b3a0c7ceafb6d5c11989b37ec33ee4a866e8: Status 404 returned error can't find the container with id a4463385ab6417b760474bca1503b3a0c7ceafb6d5c11989b37ec33ee4a866e8
Sep 08 00:02:51.445275 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.445255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbcd392_b02f_4a8b_b6b3_97713445f430.slice/crio-efc49db03f18d1a256f80a12c4dd0b3cd896527cfc343b9c1a1771b0f5805bab WatchSource:0}: Error finding container efc49db03f18d1a256f80a12c4dd0b3cd896527cfc343b9c1a1771b0f5805bab: Status 404 returned error can't find the container with id efc49db03f18d1a256f80a12c4dd0b3cd896527cfc343b9c1a1771b0f5805bab
Sep 08 00:02:51.448520 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.448451    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c37a23d_2a77_4773_af02_1ea914005127.slice/crio-38cda7c5dfdadcdd872c2c9d325bb593a2f961e3bdd0558955e69a5747ef03d8 WatchSource:0}: Error finding container 38cda7c5dfdadcdd872c2c9d325bb593a2f961e3bdd0558955e69a5747ef03d8: Status 404 returned error can't find the container with id 38cda7c5dfdadcdd872c2c9d325bb593a2f961e3bdd0558955e69a5747ef03d8
Sep 08 00:02:51.449636 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.449476    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1e5563_87ef_49d6_b328_2b208ea3da80.slice/crio-a1f55629cd1e2c5f945020db8e51e4c71b35f3c1d169c09e1f6a4ce5e57de595 WatchSource:0}: Error finding container a1f55629cd1e2c5f945020db8e51e4c71b35f3c1d169c09e1f6a4ce5e57de595: Status 404 returned error can't find the container with id a1f55629cd1e2c5f945020db8e51e4c71b35f3c1d169c09e1f6a4ce5e57de595
Sep 08 00:02:51.450834 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.450792    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed3c48f_fe87_489a_b011_76cb7b05fe30.slice/crio-422fcc63c79a306af4824c78fa41436ac62c34a193910809c4f7975468be4c42 WatchSource:0}: Error finding container 422fcc63c79a306af4824c78fa41436ac62c34a193910809c4f7975468be4c42: Status 404 returned error can't find the container with id 422fcc63c79a306af4824c78fa41436ac62c34a193910809c4f7975468be4c42
Sep 08 00:02:51.463702 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.463677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e0c506_9b9d_4525_a5e5_e30569246604.slice/crio-dc764576af4c06b9d4d33cc8ba23b22b91b5a044ac2bdae4f48adb4aa5bad047 WatchSource:0}: Error finding container dc764576af4c06b9d4d33cc8ba23b22b91b5a044ac2bdae4f48adb4aa5bad047: Status 404 returned error can't find the container with id dc764576af4c06b9d4d33cc8ba23b22b91b5a044ac2bdae4f48adb4aa5bad047
Sep 08 00:02:51.465244 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:02:51.465222    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b23f53_7ab5_4440_bb3f_9cb0b3b7e997.slice/crio-62ed5c3c39f7d46c72ea76f70542706d5c797d799d16007740da8d5cae2550f1 WatchSource:0}: Error finding container 62ed5c3c39f7d46c72ea76f70542706d5c797d799d16007740da8d5cae2550f1: Status 404 returned error can't find the container with id 62ed5c3c39f7d46c72ea76f70542706d5c797d799d16007740da8d5cae2550f1
Sep 08 00:02:51.837409 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.837118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:51.837577 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.837276    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:51.837577 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.837566    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:53.837528764 +0000 UTC m=+6.009793627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:51.938997 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.938874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:51.939167 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:51.939024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:51.939167 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939083    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:02:51.939167 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939116    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:02:51.939167 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939130    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:51.939167 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939162    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:51.939446 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:02:53.939174519 +0000 UTC m=+6.111439401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:51.939446 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:51.939215    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:53.939205688 +0000 UTC m=+6.111470557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:52.249562 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.249376    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 23:57:50 +0000 UTC" deadline="2028-02-29 23:15:01.088295188 +0000 UTC"
Sep 08 00:02:52.249562 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.249491    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12959h12m8.838809471s"
Sep 08 00:02:52.325408 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.325375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:52.325606 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:52.325527    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:02:52.326027 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.326007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:52.326155 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:52.326115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:02:52.346146 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.346064    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f47g" event={"ID":"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997","Type":"ContainerStarted","Data":"62ed5c3c39f7d46c72ea76f70542706d5c797d799d16007740da8d5cae2550f1"}
Sep 08 00:02:52.351592 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.351488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnsbf" event={"ID":"1c37a23d-2a77-4773-af02-1ea914005127","Type":"ContainerStarted","Data":"38cda7c5dfdadcdd872c2c9d325bb593a2f961e3bdd0558955e69a5747ef03d8"}
Sep 08 00:02:52.354954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.354870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ppdfm" event={"ID":"7f1e5563-87ef-49d6-b328-2b208ea3da80","Type":"ContainerStarted","Data":"a1f55629cd1e2c5f945020db8e51e4c71b35f3c1d169c09e1f6a4ce5e57de595"}
Sep 08 00:02:52.359981 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.359913    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qrfmj" event={"ID":"0cbcd392-b02f-4a8b-b6b3-97713445f430","Type":"ContainerStarted","Data":"efc49db03f18d1a256f80a12c4dd0b3cd896527cfc343b9c1a1771b0f5805bab"}
Sep 08 00:02:52.365719 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.365693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"3a35f15de2d416277d2750840c0e2994ddb33f2f01d674a1d25bb3594322fda4"}
Sep 08 00:02:52.368694 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.368622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerStarted","Data":"422fcc63c79a306af4824c78fa41436ac62c34a193910809c4f7975468be4c42"}
Sep 08 00:02:52.371493 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.371418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2" event={"ID":"10e0c506-9b9d-4525-a5e5-e30569246604","Type":"ContainerStarted","Data":"dc764576af4c06b9d4d33cc8ba23b22b91b5a044ac2bdae4f48adb4aa5bad047"}
Sep 08 00:02:52.374542 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.374498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztmxv" event={"ID":"77a53b91-7079-40c5-a9d0-1cf368aac1ef","Type":"ContainerStarted","Data":"a4463385ab6417b760474bca1503b3a0c7ceafb6d5c11989b37ec33ee4a866e8"}
Sep 08 00:02:52.377183 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.377130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j9n55" event={"ID":"583e7bd9-a4cb-4a45-9a66-255aef584166","Type":"ContainerStarted","Data":"d7a3cac751c93eeb8e055a43cd41dea4ba3853fd83599e82315dcd37c5ab371f"}
Sep 08 00:02:52.385037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.385009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal" event={"ID":"1b2683d5f56405395b041d9bac31f181","Type":"ContainerStarted","Data":"06415261229f7b3e58d42f29040f75eee2477c602435ed4136e6d684167a8994"}
Sep 08 00:02:52.392903 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.392876    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b43182a60cdc7752da5f71487404314" containerID="f8a9978883f614cfca492fda9e3ed9ccbcb99aa37f291097456c7aa582448f27" exitCode=0
Sep 08 00:02:52.392999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.392915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal" event={"ID":"6b43182a60cdc7752da5f71487404314","Type":"ContainerDied","Data":"f8a9978883f614cfca492fda9e3ed9ccbcb99aa37f291097456c7aa582448f27"}
Sep 08 00:02:52.398135 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:52.398088    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-150.ec2.internal" podStartSLOduration=2.398071073 podStartE2EDuration="2.398071073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:02:52.397770532 +0000 UTC m=+4.570035423" watchObservedRunningTime="2026-09-08 00:02:52.398071073 +0000 UTC m=+4.570335963"
Sep 08 00:02:53.324670 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:53.324572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:53.325161 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.324719    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:02:53.438943 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:53.438732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal" event={"ID":"6b43182a60cdc7752da5f71487404314","Type":"ContainerStarted","Data":"8c26093f79fd869240f377fd0f7aeb52752e01a46a1d3e8b48fc2d0760662c27"}
Sep 08 00:02:53.863672 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:53.863057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:53.863672 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.863227    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:53.863672 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.863288    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:57.863269104 +0000 UTC m=+10.035533969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:53.963872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:53.963927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964054    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964111    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:02:57.964092849 +0000 UTC m=+10.136357717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964196    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964211    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964222    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:53.964585 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:53.964258    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:02:57.964246271 +0000 UTC m=+10.136511152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:54.326828 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:54.326765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:54.327273 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:54.326893    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:02:54.327354 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:54.327335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:54.327565 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:54.327443    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:02:55.325278 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:55.324362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:55.325278 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:55.324490    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:02:56.325493 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:56.325467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:56.325954 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:56.325613    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:02:56.325954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:56.325686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:56.325954 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:56.325817    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:02:57.325047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:57.325015    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:57.325216 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.325144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:02:57.898187 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:57.898143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:57.898667 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.898297    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:57.898667 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.898366    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:05.898347869 +0000 UTC m=+18.070612757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:02:57.999384 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:57.999344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:57.999574 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999559    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:57.999717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:57.999567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:57.999717 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999634    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:05.999613566 +0000 UTC m=+18.171878438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:02:57.999831 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999727    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:02:57.999831 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999746    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:02:57.999831 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999760    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:57.999831 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:57.999812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:03:05.999796696 +0000 UTC m=+18.172061572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:02:58.326903 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:58.326180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:02:58.326903 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:58.326303    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:02:58.326903 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:58.326749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:02:58.326903 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:58.326860    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:02:59.324988 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:02:59.324660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:02:59.324988 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:02:59.324797    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:00.325480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:00.325063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:00.325480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:00.325095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:00.325480 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:00.325206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:00.325480 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:00.325464    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:01.324994 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:01.324961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:01.325188 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:01.325069    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:02.324408 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:02.324369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:02.324825 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:02.324473    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:02.324825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:02.324545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:02.324825 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:02.324671    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:03.324784 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:03.324751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:03.325193 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:03.324867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:04.325014 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:04.324971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:04.325510 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:04.325110    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:04.325510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:04.325156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:04.325510 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:04.325262    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:05.324499 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:05.324454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:05.324712 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:05.324611    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:05.957471 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:05.957385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:05.957886 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:05.957602    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:03:05.957886 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:05.957678    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:21.957658091 +0000 UTC m=+34.129922967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:03:06.057983 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:06.057938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:06.058169 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:06.058040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:06.058169 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058104    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:03:06.058283 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058167    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:03:06.058283 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058181    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:22.058159196 +0000 UTC m=+34.230424082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:03:06.058283 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058184    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:03:06.058283 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058208    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:03:06.058283 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.058245    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:03:22.058234047 +0000 UTC m=+34.230498910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:03:06.325163 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:06.325073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:06.325312 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:06.325074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:06.325312 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.325214    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:06.325312 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:06.325276    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:07.324881 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:07.324846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:07.325340 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:07.324965    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:08.324986 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:08.324946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:08.325410 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:08.325057    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:08.325410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:08.325111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:08.325410 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:08.325241    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:09.324427 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:09.324396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:09.324590 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:09.324496    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:09.477412 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:09.477372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztmxv" event={"ID":"77a53b91-7079-40c5-a9d0-1cf368aac1ef","Type":"ContainerStarted","Data":"1329ca57564a6b688a77253e6875b4f70e472da9a959355ddb2f1dbc42284a60"}
Sep 08 00:03:09.494465 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:09.493585    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-150.ec2.internal" podStartSLOduration=19.49356636 podStartE2EDuration="19.49356636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:02:53.453566575 +0000 UTC m=+5.625831455" watchObservedRunningTime="2026-09-08 00:03:09.49356636 +0000 UTC m=+21.665831244"
Sep 08 00:03:09.494465 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:09.494003    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ztmxv" podStartSLOduration=3.6003296049999998 podStartE2EDuration="21.493992829s" totalImagesPullingTime="17.893663224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:09.493132697 +0000 UTC m=+21.665397582" watchObservedRunningTime="2026-09-08 00:03:09.493992829 +0000 UTC m=+21.666257714"
Sep 08 00:03:10.327524 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.327340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:10.327769 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.327396    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:10.327769 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:10.327630    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:10.327769 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:10.327658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:10.480690 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.480657    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="8b6766a06dfa6e4af7c1bec38a0d4b7d9b12b1779f4c538e4a326cfe8a55f635" exitCode=0
Sep 08 00:03:10.481522 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.480731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"8b6766a06dfa6e4af7c1bec38a0d4b7d9b12b1779f4c538e4a326cfe8a55f635"}
Sep 08 00:03:10.482308 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.482207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2" event={"ID":"10e0c506-9b9d-4525-a5e5-e30569246604","Type":"ContainerStarted","Data":"607fd2d0003fb6b0591733f1df9cecf35d8fc45d6d5467d0bef8c6ad26f87caa"}
Sep 08 00:03:10.483747 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.483727    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j9n55" event={"ID":"583e7bd9-a4cb-4a45-9a66-255aef584166","Type":"ContainerStarted","Data":"a5fcba7bdb1e41a819804be2a27e64fc45e01c134bc7dc92a4ec0aa073e1317c"}
Sep 08 00:03:10.485292 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.485270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5f47g" event={"ID":"c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997","Type":"ContainerStarted","Data":"0a1c8ba06e721f1d6d6fe2a02ff117972ed1a49450e7a9a1bbe66f06da062409"}
Sep 08 00:03:10.486645 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.486625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnsbf" event={"ID":"1c37a23d-2a77-4773-af02-1ea914005127","Type":"ContainerStarted","Data":"c9c5632ec5351ae6ccbfaeb58588904af8a6b66c4504f2962b7d38d74db43fb4"}
Sep 08 00:03:10.488500 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.488469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ppdfm" event={"ID":"7f1e5563-87ef-49d6-b328-2b208ea3da80","Type":"ContainerStarted","Data":"41df9e92b95b5b30b45af0c0ad33b89a20191444e6ec209b1b3d04774a02f560"}
Sep 08 00:03:10.489782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.489762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qrfmj" event={"ID":"0cbcd392-b02f-4a8b-b6b3-97713445f430","Type":"ContainerStarted","Data":"5589081bdbae92e475c1480a3e6893ca4112383b4490f3fc559cfbb34b2a0b48"}
Sep 08 00:03:10.492588 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"65e8b5fe5bf4d160d02da3edd02ba7d3538f3de0f5284d5ffb45a01d6069542e"}
Sep 08 00:03:10.492662 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"2b620028ad6ac5e9e541b4fd806d9cdc8e741a9737ecf67006f3183a647c30f4"}
Sep 08 00:03:10.492662 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"a97a82d388de163ea0dd0c7b75b940786432e9040f576e47534a5597cee2d4ed"}
Sep 08 00:03:10.492729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492687    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"337171f2efc1b4dbea5f3d62499897eb6b4cf1ae1da667546f6b5ed84629bf1f"}
Sep 08 00:03:10.492762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"9e9d3c929794ed244f082aefb911b92329fd399f08e542f866dc43fa1573cfa0"}
Sep 08 00:03:10.492793 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.492787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"c342789f143e57e48a01043af11e73c7133b6ce98724070a121dbb499ff25efa"}
Sep 08 00:03:10.522169 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.522117    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5f47g" podStartSLOduration=4.649216076 podStartE2EDuration="22.522102929s" totalImagesPullingTime="17.872886853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:10.510662084 +0000 UTC m=+22.682926967" watchObservedRunningTime="2026-09-08 00:03:10.522102929 +0000 UTC m=+22.694367813"
Sep 08 00:03:10.522489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.522468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ppdfm" podStartSLOduration=4.636000114 podStartE2EDuration="22.522462087s" totalImagesPullingTime="17.886461973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:10.522027998 +0000 UTC m=+22.694292882" watchObservedRunningTime="2026-09-08 00:03:10.522462087 +0000 UTC m=+22.694726971"
Sep 08 00:03:10.547990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.547943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j9n55" podStartSLOduration=4.591260745 podStartE2EDuration="22.547928547s" totalImagesPullingTime="17.956667802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:10.547908195 +0000 UTC m=+22.720173079" watchObservedRunningTime="2026-09-08 00:03:10.547928547 +0000 UTC m=+22.720193421"
Sep 08 00:03:10.548150 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.548113    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qrfmj" podStartSLOduration=4.650991286 podStartE2EDuration="22.548108053s" totalImagesPullingTime="17.897116767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:10.53387081 +0000 UTC m=+22.706135695" watchObservedRunningTime="2026-09-08 00:03:10.548108053 +0000 UTC m=+22.720373022"
Sep 08 00:03:10.561133 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:10.561092    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnsbf" podStartSLOduration=4.651723048 podStartE2EDuration="22.561074633s" totalImagesPullingTime="17.909351585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:10.560856529 +0000 UTC m=+22.733121412" watchObservedRunningTime="2026-09-08 00:03:10.561074633 +0000 UTC m=+22.733339517"
Sep 08 00:03:11.055548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.055503    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 00:03:11.290350 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.290145    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T00:03:11.055520353Z","UUID":"e6b2dd67-db13-44b0-8670-88d7e358de69","Handler":null,"Name":"","Endpoint":""}
Sep 08 00:03:11.292128 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.292104    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 00:03:11.292244 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.292138    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 00:03:11.325029 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.325002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:11.325177 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:11.325104    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:11.496829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:11.496790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2" event={"ID":"10e0c506-9b9d-4525-a5e5-e30569246604","Type":"ContainerStarted","Data":"d1604440f993270d54a7765d88205ae140023a9d71091fcf199a7ddfd1c70fb6"}
Sep 08 00:03:12.073382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.073351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:03:12.074048 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.074023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:03:12.324481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.324408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:12.324481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.324432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:12.324707 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:12.324546    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:12.324707 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:12.324673    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:12.502240 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.502055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"df049c4ad61e20534f3bad771c11e1d60fcdc8d76790eebd09bf36e7b201213b"}
Sep 08 00:03:12.504095 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.504061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2" event={"ID":"10e0c506-9b9d-4525-a5e5-e30569246604","Type":"ContainerStarted","Data":"37c567f9d54a50156100d580cb55c30d29417db7c9438440b188d985b87d475c"}
Sep 08 00:03:12.504291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.504272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:03:12.504726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.504712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qrfmj"
Sep 08 00:03:12.518427 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:12.518381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gj2f2" podStartSLOduration=3.977878719 podStartE2EDuration="24.518368083s" totalImagesPullingTime="20.540489364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:12.517803906 +0000 UTC m=+24.690068792" watchObservedRunningTime="2026-09-08 00:03:12.518368083 +0000 UTC m=+24.690632967"
Sep 08 00:03:13.324546 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:13.324505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:13.324728 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:13.324616    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:14.325200 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:14.325161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:14.325786 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:14.325328    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:14.325786 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:14.325392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:14.325786 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:14.325493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:15.325161 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.324976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:15.325307 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:15.325263    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:15.514234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.514167    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" event={"ID":"b84de74e-3f1f-4743-920d-4f625c76a3f0","Type":"ContainerStarted","Data":"97666f29a8088ba58e0695407dd349133af0ec1604275287d169afd8022ce27b"}
Sep 08 00:03:15.514452 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.514432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:15.514568 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.514508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:15.516239 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.516199    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="672ce4b07f0317fa98f527a334dd8e8d1b868875147c39c1717447fc70e1a82c" exitCode=0
Sep 08 00:03:15.516349 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.516256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"672ce4b07f0317fa98f527a334dd8e8d1b868875147c39c1717447fc70e1a82c"}
Sep 08 00:03:15.530872 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.530849    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:15.530960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.530939    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:15.538686 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:15.538648    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn" podStartSLOduration=9.563931574 podStartE2EDuration="27.538634708s" totalImagesPullingTime="17.974703134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:15.537287917 +0000 UTC m=+27.709552802" watchObservedRunningTime="2026-09-08 00:03:15.538634708 +0000 UTC m=+27.710899593"
Sep 08 00:03:16.325153 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.324976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:16.325153 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:16.325127    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:16.325669 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.325251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:16.325669 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:16.325366    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:16.521081 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.521046    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="03c0da9269d708e1e9f917da8718925118904a763594e975aec0c146e438a8ca" exitCode=0
Sep 08 00:03:16.521264 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.521122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"03c0da9269d708e1e9f917da8718925118904a763594e975aec0c146e438a8ca"}
Sep 08 00:03:16.521264 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.521224    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 00:03:16.989066 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.989031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xpvcr"]
Sep 08 00:03:16.989246 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.989146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:16.989302 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:16.989259    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:16.992006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.991979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rf9cf"]
Sep 08 00:03:16.992147 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.992088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:16.992215 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:16.992184    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:16.992869 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.992848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-54s8c"]
Sep 08 00:03:16.992983 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:16.992933    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:16.993049 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:16.993028    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:17.523650 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:17.523617    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 00:03:18.324860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.324639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:18.325015 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.324669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:18.326097 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.326071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:18.326225 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:18.326187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:18.326559 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:18.326440    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:18.326748 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:18.326724    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:18.528251 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.528159    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="c01463f01ee1a4e35339dab1a1a953ec1088f2c754c392e1577937cde195a713" exitCode=0
Sep 08 00:03:18.528251 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.528204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"c01463f01ee1a4e35339dab1a1a953ec1088f2c754c392e1577937cde195a713"}
Sep 08 00:03:18.556943 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:18.556902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:20.324833 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:20.324796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:20.325597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:20.324795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:20.325597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:20.324914    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:20.325597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:20.324941    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:20.325597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:20.325038    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:20.325597 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:20.325110    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:21.969546 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:21.969499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:21.970039 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:21.969646    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 00:03:21.970039 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:21.969716    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret podName:b228f4d3-1a89-4fa8-87a0-78a71024d15e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:53.96969484 +0000 UTC m=+66.141959726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret") pod "global-pull-secret-syncer-xpvcr" (UID: "b228f4d3-1a89-4fa8-87a0-78a71024d15e") : object "kube-system"/"original-pull-secret" not registered
Sep 08 00:03:22.070176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:22.070142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:22.070361 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070317    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:03:22.070429 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:22.070317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:22.070429 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070388    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs podName:2ec924f8-1839-40bd-84b0-9220783b414e nodeName:}" failed. No retries permitted until 2026-09-08 00:03:54.070366881 +0000 UTC m=+66.242631743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs") pod "network-metrics-daemon-54s8c" (UID: "2ec924f8-1839-40bd-84b0-9220783b414e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 00:03:22.070429 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070422    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 00:03:22.070614 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070443    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 00:03:22.070614 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070456    2580 projected.go:196] Error preparing data for projected volume kube-api-access-vv6r2 for pod openshift-network-diagnostics/network-check-target-rf9cf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:03:22.070614 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.070511    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2 podName:4ca3d68b-7b7e-46d5-86cf-228ca1312423 nodeName:}" failed. No retries permitted until 2026-09-08 00:03:54.070494176 +0000 UTC m=+66.242759060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vv6r2" (UniqueName: "kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2") pod "network-check-target-rf9cf" (UID: "4ca3d68b-7b7e-46d5-86cf-228ca1312423") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 00:03:22.329157 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:22.329128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:22.329157 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:22.329144    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:22.329442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:22.329128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:22.329442 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.329239    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xpvcr" podUID="b228f4d3-1a89-4fa8-87a0-78a71024d15e"
Sep 08 00:03:22.329442 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.329328    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rf9cf" podUID="4ca3d68b-7b7e-46d5-86cf-228ca1312423"
Sep 08 00:03:22.329442 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:22.329426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-54s8c" podUID="2ec924f8-1839-40bd-84b0-9220783b414e"
Sep 08 00:03:23.070425 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.070394    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-150.ec2.internal" event="NodeReady"
Sep 08 00:03:23.070970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.070578    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 00:03:23.109893 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.109860    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nshh7"]
Sep 08 00:03:23.144573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.144475    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v4kjq"]
Sep 08 00:03:23.144795 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.144573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.146824 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.146802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 00:03:23.146933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.146823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xbmqx\""
Sep 08 00:03:23.146933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.146809    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 00:03:23.158082 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.158008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nshh7"]
Sep 08 00:03:23.158082 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.158053    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4kjq"]
Sep 08 00:03:23.158231 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.158089    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tdw48"]
Sep 08 00:03:23.158231 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.158192    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.160090 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.160071    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8cwdj\""
Sep 08 00:03:23.160241 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.160219    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 00:03:23.160315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.160234    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 00:03:23.160448 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.160428    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 00:03:23.160559 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.160491    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 00:03:23.176135 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.176047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tdw48"]
Sep 08 00:03:23.176246 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.176156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.178206 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.178163    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 00:03:23.178206 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.178194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 00:03:23.178379 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.178276    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9vvpw\""
Sep 08 00:03:23.178438 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.178426    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 00:03:23.283102 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzp8\" (UniqueName: \"kubernetes.io/projected/57937680-0364-41f0-b9ae-39a9d7c0a1cd-kube-api-access-6mzp8\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.283279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zz7\" (UniqueName: \"kubernetes.io/projected/c0fd4558-3d7c-44d3-b67d-19c541d6b714-kube-api-access-c2zz7\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.283279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.283279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r7j\" (UniqueName: \"kubernetes.io/projected/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-api-access-l8r7j\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.283279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.283279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57937680-0364-41f0-b9ae-39a9d7c0a1cd-config-volume\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.283493 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0fd4558-3d7c-44d3-b67d-19c541d6b714-cert\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.283554 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-crio-socket\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.283612 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-data-volume\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.283612 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.283698 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.283655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937680-0364-41f0-b9ae-39a9d7c0a1cd-tmp-dir\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.384448 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57937680-0364-41f0-b9ae-39a9d7c0a1cd-config-volume\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.384448 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0fd4558-3d7c-44d3-b67d-19c541d6b714-cert\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.384702 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-crio-socket\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.384702 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-data-volume\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.384702 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.384702 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.384664    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:23.384877 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.384730    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:23.884711024 +0000 UTC m=+36.056975887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:23.384877 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-crio-socket\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.384990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937680-0364-41f0-b9ae-39a9d7c0a1cd-tmp-dir\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.384990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-data-volume\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.384990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzp8\" (UniqueName: \"kubernetes.io/projected/57937680-0364-41f0-b9ae-39a9d7c0a1cd-kube-api-access-6mzp8\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.384990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.384971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zz7\" (UniqueName: \"kubernetes.io/projected/c0fd4558-3d7c-44d3-b67d-19c541d6b714-kube-api-access-c2zz7\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.385184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.385011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.385184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.385050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r7j\" (UniqueName: \"kubernetes.io/projected/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-api-access-l8r7j\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.385184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.385080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.385184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.385144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937680-0364-41f0-b9ae-39a9d7c0a1cd-tmp-dir\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.385184 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.385156    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:03:23.385399 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.385205    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:03:23.385399 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.385224    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls podName:57937680-0364-41f0-b9ae-39a9d7c0a1cd nodeName:}" failed. No retries permitted until 2026-09-08 00:03:23.885207947 +0000 UTC m=+36.057472837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls") pod "dns-default-nshh7" (UID: "57937680-0364-41f0-b9ae-39a9d7c0a1cd") : secret "dns-default-metrics-tls" not found
Sep 08 00:03:23.385399 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.385230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57937680-0364-41f0-b9ae-39a9d7c0a1cd-config-volume\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.385399 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.385280    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:23.8852564 +0000 UTC m=+36.057521288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : configmap "kube-rbac-proxy" not found
Sep 08 00:03:23.389119 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.389090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0fd4558-3d7c-44d3-b67d-19c541d6b714-cert\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.400687 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.400624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzp8\" (UniqueName: \"kubernetes.io/projected/57937680-0364-41f0-b9ae-39a9d7c0a1cd-kube-api-access-6mzp8\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.400687 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.400626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zz7\" (UniqueName: \"kubernetes.io/projected/c0fd4558-3d7c-44d3-b67d-19c541d6b714-kube-api-access-c2zz7\") pod \"ingress-canary-tdw48\" (UID: \"c0fd4558-3d7c-44d3-b67d-19c541d6b714\") " pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.400853 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.400697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r7j\" (UniqueName: \"kubernetes.io/projected/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-api-access-l8r7j\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.486606 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.486571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tdw48"
Sep 08 00:03:23.891208 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.890971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.891393 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891120    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:23.891393 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891337    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:24.891318212 +0000 UTC m=+37.063583090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:23.891393 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891347    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:03:23.891393 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891378    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls podName:57937680-0364-41f0-b9ae-39a9d7c0a1cd nodeName:}" failed. No retries permitted until 2026-09-08 00:03:24.891368679 +0000 UTC m=+37.063633547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls") pod "dns-default-nshh7" (UID: "57937680-0364-41f0-b9ae-39a9d7c0a1cd") : secret "dns-default-metrics-tls" not found
Sep 08 00:03:23.891393 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.891268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:23.891620 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:23.891419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:23.891620 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891485    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:03:23.891620 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:23.891522    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:24.891512545 +0000 UTC m=+37.063777421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : configmap "kube-rbac-proxy" not found
Sep 08 00:03:24.014766 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.014666    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tdw48"]
Sep 08 00:03:24.018268 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:24.018242    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fd4558_3d7c_44d3_b67d_19c541d6b714.slice/crio-aa1dfa0ef01631e56cd50a7116de3c3d18de938f34ffb7b9bacdf1884f047dcb WatchSource:0}: Error finding container aa1dfa0ef01631e56cd50a7116de3c3d18de938f34ffb7b9bacdf1884f047dcb: Status 404 returned error can't find the container with id aa1dfa0ef01631e56cd50a7116de3c3d18de938f34ffb7b9bacdf1884f047dcb
Sep 08 00:03:24.286728 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.286703    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f47g_c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997/dns-node-resolver/0.log"
Sep 08 00:03:24.324386 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.324353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:24.324715 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.324462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:24.324715 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.324681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:24.326644 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.326621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 00:03:24.326757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.326621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7g2f\""
Sep 08 00:03:24.326757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.326629    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 00:03:24.327354 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.327325    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 00:03:24.327354 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.327343    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgbhs\""
Sep 08 00:03:24.327506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.327375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 00:03:24.479743 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.479705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"]
Sep 08 00:03:24.489649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.489621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"
Sep 08 00:03:24.491130 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.491101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"]
Sep 08 00:03:24.492080 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.491827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6vrt7\""
Sep 08 00:03:24.492080 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.491880    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 00:03:24.492237 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.492107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 00:03:24.544922 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.544830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tdw48" event={"ID":"c0fd4558-3d7c-44d3-b67d-19c541d6b714","Type":"ContainerStarted","Data":"aa1dfa0ef01631e56cd50a7116de3c3d18de938f34ffb7b9bacdf1884f047dcb"}
Sep 08 00:03:24.548040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.548016    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="f60c166f7360dea25a8c3a413eb4be562834d00d9f153ccefded64401d0e3dca" exitCode=0
Sep 08 00:03:24.548175 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.548051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"f60c166f7360dea25a8c3a413eb4be562834d00d9f153ccefded64401d0e3dca"}
Sep 08 00:03:24.598255 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.598218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvf2\" (UniqueName: \"kubernetes.io/projected/997ae1d7-00b3-4316-83a0-f116a4e93eae-kube-api-access-tmvf2\") pod \"migrator-fdbb8fdff-fn8qz\" (UID: \"997ae1d7-00b3-4316-83a0-f116a4e93eae\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"
Sep 08 00:03:24.698841 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.698818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvf2\" (UniqueName: \"kubernetes.io/projected/997ae1d7-00b3-4316-83a0-f116a4e93eae-kube-api-access-tmvf2\") pod \"migrator-fdbb8fdff-fn8qz\" (UID: \"997ae1d7-00b3-4316-83a0-f116a4e93eae\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"
Sep 08 00:03:24.706729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.706708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvf2\" (UniqueName: \"kubernetes.io/projected/997ae1d7-00b3-4316-83a0-f116a4e93eae-kube-api-access-tmvf2\") pod \"migrator-fdbb8fdff-fn8qz\" (UID: \"997ae1d7-00b3-4316-83a0-f116a4e93eae\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"
Sep 08 00:03:24.801908 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.801835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"
Sep 08 00:03:24.901233 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.901017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:24.901411 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.901283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:24.901411 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901141    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:03:24.901411 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:24.901367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:24.901411 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901384    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:26.901361994 +0000 UTC m=+39.073626870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : configmap "kube-rbac-proxy" not found
Sep 08 00:03:24.901633 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901456    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:03:24.901633 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901464    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:24.901633 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901492    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls podName:57937680-0364-41f0-b9ae-39a9d7c0a1cd nodeName:}" failed. No retries permitted until 2026-09-08 00:03:26.901483287 +0000 UTC m=+39.073748162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls") pod "dns-default-nshh7" (UID: "57937680-0364-41f0-b9ae-39a9d7c0a1cd") : secret "dns-default-metrics-tls" not found
Sep 08 00:03:24.901633 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:24.901521    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:26.901503597 +0000 UTC m=+39.073768474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:25.084951 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:25.084824    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz"]
Sep 08 00:03:25.089178 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:25.089150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997ae1d7_00b3_4316_83a0_f116a4e93eae.slice/crio-6ede5bfd268308d64508b5e9c0163fbc14f4529b68cdd4f852a68b2e5212580c WatchSource:0}: Error finding container 6ede5bfd268308d64508b5e9c0163fbc14f4529b68cdd4f852a68b2e5212580c: Status 404 returned error can't find the container with id 6ede5bfd268308d64508b5e9c0163fbc14f4529b68cdd4f852a68b2e5212580c
Sep 08 00:03:25.553470 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:25.553437    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ed3c48f-fe87-489a-b011-76cb7b05fe30" containerID="11fab691e8644221c85286f742e119e35b024d613bde964d90666d2e7aacb981" exitCode=0
Sep 08 00:03:25.553865 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:25.553515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerDied","Data":"11fab691e8644221c85286f742e119e35b024d613bde964d90666d2e7aacb981"}
Sep 08 00:03:25.554661 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:25.554640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz" event={"ID":"997ae1d7-00b3-4316-83a0-f116a4e93eae","Type":"ContainerStarted","Data":"6ede5bfd268308d64508b5e9c0163fbc14f4529b68cdd4f852a68b2e5212580c"}
Sep 08 00:03:25.660435 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:25.660405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ppdfm_7f1e5563-87ef-49d6-b328-2b208ea3da80/node-ca/0.log"
Sep 08 00:03:26.560286 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:26.560254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" event={"ID":"4ed3c48f-fe87-489a-b011-76cb7b05fe30","Type":"ContainerStarted","Data":"9120cadefffd1934ba48ed46e4e7bea5b6233b5ca6d839e8916af433c67cc932"}
Sep 08 00:03:26.581837 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:26.581787    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fc9f2" podStartSLOduration=8.031713728 podStartE2EDuration="38.581771103s" totalImagesPullingTime="30.550057375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:26.581285208 +0000 UTC m=+38.753550095" watchObservedRunningTime="2026-09-08 00:03:26.581771103 +0000 UTC m=+38.754035986"
Sep 08 00:03:26.918337 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:26.918299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:26.918349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:26.918399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918451    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918495    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918516    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls podName:57937680-0364-41f0-b9ae-39a9d7c0a1cd nodeName:}" failed. No retries permitted until 2026-09-08 00:03:30.918498908 +0000 UTC m=+43.090763770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls") pod "dns-default-nshh7" (UID: "57937680-0364-41f0-b9ae-39a9d7c0a1cd") : secret "dns-default-metrics-tls" not found
Sep 08 00:03:26.918558 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918550    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:26.918786 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918570    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:30.918553458 +0000 UTC m=+43.090818337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : configmap "kube-rbac-proxy" not found
Sep 08 00:03:26.918786 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:26.918593    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:30.918580574 +0000 UTC m=+43.090845436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:27.564467 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:27.564429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tdw48" event={"ID":"c0fd4558-3d7c-44d3-b67d-19c541d6b714","Type":"ContainerStarted","Data":"cd9d41790b8295c0a8f97f5923ce053a5c5f4ae3b63422a5e6bfcc50765f0291"}
Sep 08 00:03:27.578317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:27.578165    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tdw48" podStartSLOduration=2.068429963 podStartE2EDuration="4.578146329s" totalImagesPullingTime="2.509716366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:27.577735221 +0000 UTC m=+39.750000107" watchObservedRunningTime="2026-09-08 00:03:27.578146329 +0000 UTC m=+39.750411252"
Sep 08 00:03:28.568569 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:28.568510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz" event={"ID":"997ae1d7-00b3-4316-83a0-f116a4e93eae","Type":"ContainerStarted","Data":"9fd6e7216449014c39f41c1b4f40496494fab585ce6f9141c7cdb2cee4644b3b"}
Sep 08 00:03:28.569003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:28.568588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz" event={"ID":"997ae1d7-00b3-4316-83a0-f116a4e93eae","Type":"ContainerStarted","Data":"39700307d1d068fdf4164bcf300332b74d966e804f04ca50d24695c5b103649c"}
Sep 08 00:03:28.581870 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:28.581819    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fn8qz" podStartSLOduration=1.971362279 podStartE2EDuration="4.581804751s" totalImagesPullingTime="2.610442472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:28.580980822 +0000 UTC m=+40.753245709" watchObservedRunningTime="2026-09-08 00:03:28.581804751 +0000 UTC m=+40.754069662"
Sep 08 00:03:30.947406 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:30.947366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:30.947437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:30.947456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:30.947565    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:30.947588    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:30.947652    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls podName:57937680-0364-41f0-b9ae-39a9d7c0a1cd nodeName:}" failed. No retries permitted until 2026-09-08 00:03:38.947630463 +0000 UTC m=+51.119895325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls") pod "dns-default-nshh7" (UID: "57937680-0364-41f0-b9ae-39a9d7c0a1cd") : secret "dns-default-metrics-tls" not found
Sep 08 00:03:30.947856 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:03:30.947674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls podName:37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de nodeName:}" failed. No retries permitted until 2026-09-08 00:03:38.947665268 +0000 UTC m=+51.119930168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v4kjq" (UID: "37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de") : secret "insights-runtime-extractor-tls" not found
Sep 08 00:03:30.948053 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:30.947951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:39.004347 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.004306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:39.004808 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.004380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:39.007665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.007639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v4kjq\" (UID: \"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de\") " pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:39.007774 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.007645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57937680-0364-41f0-b9ae-39a9d7c0a1cd-metrics-tls\") pod \"dns-default-nshh7\" (UID: \"57937680-0364-41f0-b9ae-39a9d7c0a1cd\") " pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:39.056016 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.055979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:39.067795 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.067776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v4kjq"
Sep 08 00:03:39.346809 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.346733    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nshh7"]
Sep 08 00:03:39.370771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.370742    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v4kjq"]
Sep 08 00:03:39.374007 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:39.373979    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a01fa5_4cf7_4d65_ab7b_a1b1bc5726de.slice/crio-618d2be9bec21083ec928fe88ee2866ba46797432bbbd17e6276f47782564520 WatchSource:0}: Error finding container 618d2be9bec21083ec928fe88ee2866ba46797432bbbd17e6276f47782564520: Status 404 returned error can't find the container with id 618d2be9bec21083ec928fe88ee2866ba46797432bbbd17e6276f47782564520
Sep 08 00:03:39.597324 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.597221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nshh7" event={"ID":"57937680-0364-41f0-b9ae-39a9d7c0a1cd","Type":"ContainerStarted","Data":"4a52895e0f775565f7840917b1d47398384ca0507e85609edf575a3d6365fbec"}
Sep 08 00:03:39.598692 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.598667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4kjq" event={"ID":"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de","Type":"ContainerStarted","Data":"5ab15e306cda615d896fd808c173347f74da003775cbe0117ffe69aba5d2af4c"}
Sep 08 00:03:39.598788 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:39.598715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4kjq" event={"ID":"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de","Type":"ContainerStarted","Data":"618d2be9bec21083ec928fe88ee2866ba46797432bbbd17e6276f47782564520"}
Sep 08 00:03:41.608079 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:41.607986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nshh7" event={"ID":"57937680-0364-41f0-b9ae-39a9d7c0a1cd","Type":"ContainerStarted","Data":"9176af5cb8cfc07d4294bc986730d30454a9a08bb14a9a78fe097ff8dc8b1a9b"}
Sep 08 00:03:41.608079 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:41.608062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nshh7" event={"ID":"57937680-0364-41f0-b9ae-39a9d7c0a1cd","Type":"ContainerStarted","Data":"8ca142db92a68819277415229b0fcbe14b09378cf0c425eb05f157998083dd4d"}
Sep 08 00:03:41.608574 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:41.608095    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:41.610102 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:41.610070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4kjq" event={"ID":"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de","Type":"ContainerStarted","Data":"a308f3f96c0f18486e4ed1570eae81e976ea6b78af552376f41fb5c534f66335"}
Sep 08 00:03:41.623086 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:41.623030    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nshh7" podStartSLOduration=16.708682279 podStartE2EDuration="18.623015497s" totalImagesPullingTime="1.914333218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:41.621698699 +0000 UTC m=+53.793963583" watchObservedRunningTime="2026-09-08 00:03:41.623015497 +0000 UTC m=+53.795280381"
Sep 08 00:03:42.615677 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:42.615640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v4kjq" event={"ID":"37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de","Type":"ContainerStarted","Data":"701d88e4380284e975247c47e4c2e90008d438a17ca251f3a61360f43049bfd1"}
Sep 08 00:03:42.632850 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:42.632800    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v4kjq" podStartSLOduration=17.030079818 podStartE2EDuration="19.632783525s" totalImagesPullingTime="2.602703707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:03:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:42.631625636 +0000 UTC m=+54.803890520" watchObservedRunningTime="2026-09-08 00:03:42.632783525 +0000 UTC m=+54.805048409"
Sep 08 00:03:47.611344 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.611309    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"]
Sep 08 00:03:47.614035 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.614014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.616289 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.616255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 00:03:47.616418 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.616334    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 00:03:47.616561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.616540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 00:03:47.616665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.616565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 00:03:47.617235 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.617217    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 00:03:47.617415 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.617400    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 00:03:47.617455 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.617440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 00:03:47.629210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.629186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"]
Sep 08 00:03:47.665900 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.665871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.666065 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.665906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fps4z\" (UniqueName: \"kubernetes.io/projected/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-kube-api-access-fps4z\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.666065 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.665943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.666065 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.665977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.666190 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.666059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.666190 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.666099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.711130 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.711090    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-665fcd494-75bck"]
Sep 08 00:03:47.714181 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.714148    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.717132 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.717104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 00:03:47.717279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.717198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hwm5h\""
Sep 08 00:03:47.717279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.717251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 00:03:47.717499 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.717480    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 00:03:47.721359 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.721340    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 00:03:47.727148 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.727117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-665fcd494-75bck"]
Sep 08 00:03:47.766801 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.766801 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-trusted-ca\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkxb\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-kube-api-access-hgkxb\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-bound-sa-token\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fps4z\" (UniqueName: \"kubernetes.io/projected/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-kube-api-access-fps4z\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.767010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.766991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-image-registry-private-configuration\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-tls\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-certificates\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-ca-trust-extracted\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-installation-pull-secrets\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.767584 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.767523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.769457 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.769432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-ca\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.769596 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.769496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.769759 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.769740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.769759 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.769749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.778129 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.778108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fps4z\" (UniqueName: \"kubernetes.io/projected/eeb4b9ff-ff1c-49dc-b34f-17d3363719d8-kube-api-access-fps4z\") pod \"cluster-proxy-proxy-agent-75684c894f-b5vhn\" (UID: \"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:47.868396 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-ca-trust-extracted\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868396 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-installation-pull-secrets\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-trusted-ca\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkxb\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-kube-api-access-hgkxb\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-bound-sa-token\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-image-registry-private-configuration\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868730    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-ca-trust-extracted\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868915 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-tls\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.868915 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.868792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-certificates\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.869450 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.869415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-certificates\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.869562 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.869481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-trusted-ca\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.871008 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.870981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-registry-tls\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.871172 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.871154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-installation-pull-secrets\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.871224 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.871182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-image-registry-private-configuration\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.876952 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.876928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-bound-sa-token\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.876952 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.876946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkxb\" (UniqueName: \"kubernetes.io/projected/fdd3e2b9-1d73-4933-930a-c56c09b2e5e7-kube-api-access-hgkxb\") pod \"image-registry-665fcd494-75bck\" (UID: \"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7\") " pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:47.934130 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:47.934098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"
Sep 08 00:03:48.023550 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.023493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:48.202597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.202563    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn"]
Sep 08 00:03:48.207070 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:48.207046    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeb4b9ff_ff1c_49dc_b34f_17d3363719d8.slice/crio-9a95fcbe491303b0e4bad236741953117dac66a75c6e6930c984749a08839a0f WatchSource:0}: Error finding container 9a95fcbe491303b0e4bad236741953117dac66a75c6e6930c984749a08839a0f: Status 404 returned error can't find the container with id 9a95fcbe491303b0e4bad236741953117dac66a75c6e6930c984749a08839a0f
Sep 08 00:03:48.288300 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.288270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-665fcd494-75bck"]
Sep 08 00:03:48.291353 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:48.291312    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd3e2b9_1d73_4933_930a_c56c09b2e5e7.slice/crio-4e7978ff739f052a0461bce0b11efa9c2ca11bd80a3e225db374d1b60b2c3df9 WatchSource:0}: Error finding container 4e7978ff739f052a0461bce0b11efa9c2ca11bd80a3e225db374d1b60b2c3df9: Status 404 returned error can't find the container with id 4e7978ff739f052a0461bce0b11efa9c2ca11bd80a3e225db374d1b60b2c3df9
Sep 08 00:03:48.568866 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.568787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-llnkn"
Sep 08 00:03:48.635109 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.635070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn" event={"ID":"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8","Type":"ContainerStarted","Data":"9a95fcbe491303b0e4bad236741953117dac66a75c6e6930c984749a08839a0f"}
Sep 08 00:03:48.636651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.636625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-665fcd494-75bck" event={"ID":"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7","Type":"ContainerStarted","Data":"817f97b29aea44e2c5c28eb1a4141d51bb5d77a5210ed90f5138dfd777f0593d"}
Sep 08 00:03:48.636757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.636705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-665fcd494-75bck" event={"ID":"fdd3e2b9-1d73-4933-930a-c56c09b2e5e7","Type":"ContainerStarted","Data":"4e7978ff739f052a0461bce0b11efa9c2ca11bd80a3e225db374d1b60b2c3df9"}
Sep 08 00:03:48.636757 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.636749    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:03:48.654764 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:48.654705    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-665fcd494-75bck" podStartSLOduration=1.654689389 podStartE2EDuration="1.654689389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:48.653822899 +0000 UTC m=+60.826087795" watchObservedRunningTime="2026-09-08 00:03:48.654689389 +0000 UTC m=+60.826954272"
Sep 08 00:03:51.618424 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:51.618395    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nshh7"
Sep 08 00:03:54.017003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.016955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:54.019383 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.019363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 00:03:54.030422 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.030398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b228f4d3-1a89-4fa8-87a0-78a71024d15e-original-pull-secret\") pod \"global-pull-secret-syncer-xpvcr\" (UID: \"b228f4d3-1a89-4fa8-87a0-78a71024d15e\") " pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:54.044716 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.044681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xpvcr"
Sep 08 00:03:54.117958 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.117918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:54.118129 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.118065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:54.120441 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.120423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 00:03:54.120944 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.120930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 00:03:54.131461 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.131434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ec924f8-1839-40bd-84b0-9220783b414e-metrics-certs\") pod \"network-metrics-daemon-54s8c\" (UID: \"2ec924f8-1839-40bd-84b0-9220783b414e\") " pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:54.131718 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.131699    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 00:03:54.142634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.142312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6r2\" (UniqueName: \"kubernetes.io/projected/4ca3d68b-7b7e-46d5-86cf-228ca1312423-kube-api-access-vv6r2\") pod \"network-check-target-rf9cf\" (UID: \"4ca3d68b-7b7e-46d5-86cf-228ca1312423\") " pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:54.336319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.336256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7g2f\""
Sep 08 00:03:54.341886 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.341869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgbhs\""
Sep 08 00:03:54.344956 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.344882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-54s8c"
Sep 08 00:03:54.345142 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.345123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xpvcr"]
Sep 08 00:03:54.348378 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:54.348351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228f4d3_1a89_4fa8_87a0_78a71024d15e.slice/crio-2561b7f31f243fe3431cd879419d011c0ae9af7bcd63b7977b95193075a00565 WatchSource:0}: Error finding container 2561b7f31f243fe3431cd879419d011c0ae9af7bcd63b7977b95193075a00565: Status 404 returned error can't find the container with id 2561b7f31f243fe3431cd879419d011c0ae9af7bcd63b7977b95193075a00565
Sep 08 00:03:54.349975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.349957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:03:54.636154 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.636082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-54s8c"]
Sep 08 00:03:54.639185 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:54.639157    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec924f8_1839_40bd_84b0_9220783b414e.slice/crio-7adf8d31daf97825c90a5ab37cbd934a8627da92644c2ec6f765f97d32d1ee5c WatchSource:0}: Error finding container 7adf8d31daf97825c90a5ab37cbd934a8627da92644c2ec6f765f97d32d1ee5c: Status 404 returned error can't find the container with id 7adf8d31daf97825c90a5ab37cbd934a8627da92644c2ec6f765f97d32d1ee5c
Sep 08 00:03:54.657182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.657155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xpvcr" event={"ID":"b228f4d3-1a89-4fa8-87a0-78a71024d15e","Type":"ContainerStarted","Data":"2561b7f31f243fe3431cd879419d011c0ae9af7bcd63b7977b95193075a00565"}
Sep 08 00:03:54.658387 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.658365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54s8c" event={"ID":"2ec924f8-1839-40bd-84b0-9220783b414e","Type":"ContainerStarted","Data":"7adf8d31daf97825c90a5ab37cbd934a8627da92644c2ec6f765f97d32d1ee5c"}
Sep 08 00:03:54.670978 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:54.670954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rf9cf"]
Sep 08 00:03:54.674067 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:03:54.674045    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca3d68b_7b7e_46d5_86cf_228ca1312423.slice/crio-382399140ff3d68468f67acc661db511cdb2db5b4e217b94916ccfa72a0e6b92 WatchSource:0}: Error finding container 382399140ff3d68468f67acc661db511cdb2db5b4e217b94916ccfa72a0e6b92: Status 404 returned error can't find the container with id 382399140ff3d68468f67acc661db511cdb2db5b4e217b94916ccfa72a0e6b92
Sep 08 00:03:55.665057 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:55.664992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rf9cf" event={"ID":"4ca3d68b-7b7e-46d5-86cf-228ca1312423","Type":"ContainerStarted","Data":"382399140ff3d68468f67acc661db511cdb2db5b4e217b94916ccfa72a0e6b92"}
Sep 08 00:03:57.676320 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:57.676271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54s8c" event={"ID":"2ec924f8-1839-40bd-84b0-9220783b414e","Type":"ContainerStarted","Data":"194d87ca296ae8d1281a9edece16d1d1461976fa632ef5f01f9c18666aeb0109"}
Sep 08 00:03:57.676812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:57.676341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-54s8c" event={"ID":"2ec924f8-1839-40bd-84b0-9220783b414e","Type":"ContainerStarted","Data":"37be785928238bce808ab70477f7d22e458373ae3deaeadb4c5747ffcb47a031"}
Sep 08 00:03:57.691457 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:03:57.691403    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-54s8c" podStartSLOduration=67.1742935 podStartE2EDuration="1m9.69138132s" totalImagesPullingTime="2.51708782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:03:57.69039238 +0000 UTC m=+69.862657265" watchObservedRunningTime="2026-09-08 00:03:57.69138132 +0000 UTC m=+69.863646204"
Sep 08 00:04:01.693265 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.693229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rf9cf" event={"ID":"4ca3d68b-7b7e-46d5-86cf-228ca1312423","Type":"ContainerStarted","Data":"6b0aab0454c64887ceacc2e8faefea5622ad693e36717ecf7f67fb8bf4ebb105"}
Sep 08 00:04:01.693763 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.693359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:04:01.694804 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.694783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xpvcr" event={"ID":"b228f4d3-1a89-4fa8-87a0-78a71024d15e","Type":"ContainerStarted","Data":"286aef362570dc5f5880bf0808ad9b0697ef357eff71c9818522b4e4a496fd1a"}
Sep 08 00:04:01.696823 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.696801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn" event={"ID":"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8","Type":"ContainerStarted","Data":"84a18c08c2b690d8d933f40c79297bef3d950776edc871157d5c9cbc2a9f6474"}
Sep 08 00:04:01.696909 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.696833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn" event={"ID":"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8","Type":"ContainerStarted","Data":"2f58a6a093529e17fba75ec7368cdecd8e3b2f0c9798d344b47fff861265d6b6"}
Sep 08 00:04:01.696909 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.696852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn" event={"ID":"eeb4b9ff-ff1c-49dc-b34f-17d3363719d8","Type":"ContainerStarted","Data":"0649d327fd9788dc6bbd93246ff687560de74007659cb977c30cd3e9573d1b7f"}
Sep 08 00:04:01.707121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.706701    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rf9cf" podStartSLOduration=67.317818782 podStartE2EDuration="1m13.706685134s" totalImagesPullingTime="6.388866352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:01.706223563 +0000 UTC m=+73.878488447" watchObservedRunningTime="2026-09-08 00:04:01.706685134 +0000 UTC m=+73.878950019"
Sep 08 00:04:01.722678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.722636    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75684c894f-b5vhn" podStartSLOduration=1.8607657199999998 podStartE2EDuration="14.722622477s" totalImagesPullingTime="12.861856757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:01.721945774 +0000 UTC m=+73.894210659" watchObservedRunningTime="2026-09-08 00:04:01.722622477 +0000 UTC m=+73.894887361"
Sep 08 00:04:01.735356 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.735309    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xpvcr" podStartSLOduration=67.034723336 podStartE2EDuration="1m13.735294307s" totalImagesPullingTime="6.700570971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:01.735081932 +0000 UTC m=+73.907346813" watchObservedRunningTime="2026-09-08 00:04:01.735294307 +0000 UTC m=+73.907559192"
Sep 08 00:04:01.893167 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.893133    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"]
Sep 08 00:04:01.895223 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.895206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:01.897348 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 00:04:01.897348 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897331    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 00:04:01.897558 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897413    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dn4cc\""
Sep 08 00:04:01.897558 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 00:04:01.897767 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 00:04:01.897845 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.897767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 00:04:01.906084 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.906062    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"]
Sep 08 00:04:01.913704 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.913676    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"]
Sep 08 00:04:01.917248 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.917225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.919726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.919698    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mzbdq\""
Sep 08 00:04:01.919836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.919736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 00:04:01.919836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.919801    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 00:04:01.919836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.919805    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 00:04:01.921632 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.921615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ld88q"]
Sep 08 00:04:01.923678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.923661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.925826 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.925805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 00:04:01.925930 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.925844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7xzz2\""
Sep 08 00:04:01.925930 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.925853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 00:04:01.925930 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.925806    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 00:04:01.928321 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.928299    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"]
Sep 08 00:04:01.984480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-root\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.984480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-wtmp\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-metrics-client-ca\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e69cd725-f8e7-42c0-8938-9c2a1a119e64-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-sys\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bhj\" (UniqueName: \"kubernetes.io/projected/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-kube-api-access-76bhj\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-textfile\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:01.984717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwhr\" (UniqueName: \"kubernetes.io/projected/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-api-access-cgwhr\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984802    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjrx\" (UniqueName: \"kubernetes.io/projected/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-kube-api-access-9vjrx\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:01.985021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:01.984892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.085469 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-root\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085469 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-wtmp\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-metrics-client-ca\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e69cd725-f8e7-42c0-8938-9c2a1a119e64-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-sys\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-root\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:04:02.085615    2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bhj\" (UniqueName: \"kubernetes.io/projected/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-kube-api-access-76bhj\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-sys\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-textfile\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:04:02.085695    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls podName:e69cd725-f8e7-42c0-8938-9c2a1a119e64 nodeName:}" failed. No retries permitted until 2026-09-08 00:04:02.585674399 +0000 UTC m=+74.757939260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-dpdvg" (UID: "e69cd725-f8e7-42c0-8938-9c2a1a119e64") : secret "kube-state-metrics-tls" not found
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:04:02.085710    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 00:04:02.085782 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:04:02.085784    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls podName:dfe8f8b5-d294-47ae-8592-40ec7f2b6989 nodeName:}" failed. No retries permitted until 2026-09-08 00:04:02.585766845 +0000 UTC m=+74.758031707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls") pod "node-exporter-ld88q" (UID: "dfe8f8b5-d294-47ae-8592-40ec7f2b6989") : secret "node-exporter-tls" not found
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-wtmp\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwhr\" (UniqueName: \"kubernetes.io/projected/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-api-access-cgwhr\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.085985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjrx\" (UniqueName: \"kubernetes.io/projected/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-kube-api-access-9vjrx\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-textfile\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e69cd725-f8e7-42c0-8938-9c2a1a119e64-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-metrics-client-ca\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.086489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.087132 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.086670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e69cd725-f8e7-42c0-8938-9c2a1a119e64-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.087211 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.087190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.088758 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.088735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.088859 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.088761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.088899 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.088871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.088933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.088914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.095774 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.095753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwhr\" (UniqueName: \"kubernetes.io/projected/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-api-access-cgwhr\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.096744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.096726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bhj\" (UniqueName: \"kubernetes.io/projected/060c7c6e-871d-4a3f-b359-0e97a3c12aa0-kube-api-access-76bhj\") pod \"openshift-state-metrics-657fd6c54-kscxq\" (UID: \"060c7c6e-871d-4a3f-b359-0e97a3c12aa0\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.097079 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.097063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjrx\" (UniqueName: \"kubernetes.io/projected/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-kube-api-access-9vjrx\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.203419 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.203385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"
Sep 08 00:04:02.473104 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.473070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq"]
Sep 08 00:04:02.476325 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:04:02.476295    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060c7c6e_871d_4a3f_b359_0e97a3c12aa0.slice/crio-9243718fc633fffb93bc82f382804f2aba0d190490b7f436cdb949914de96bec WatchSource:0}: Error finding container 9243718fc633fffb93bc82f382804f2aba0d190490b7f436cdb949914de96bec: Status 404 returned error can't find the container with id 9243718fc633fffb93bc82f382804f2aba0d190490b7f436cdb949914de96bec
Sep 08 00:04:02.589799 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.589776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.589891 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.589812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.592232 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.592207    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dfe8f8b5-d294-47ae-8592-40ec7f2b6989-node-exporter-tls\") pod \"node-exporter-ld88q\" (UID: \"dfe8f8b5-d294-47ae-8592-40ec7f2b6989\") " pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.592310 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.592272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e69cd725-f8e7-42c0-8938-9c2a1a119e64-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-dpdvg\" (UID: \"e69cd725-f8e7-42c0-8938-9c2a1a119e64\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.702426 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.702388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq" event={"ID":"060c7c6e-871d-4a3f-b359-0e97a3c12aa0","Type":"ContainerStarted","Data":"560103277b792453bd157e8530fefac6e5a92d01084fe4873f8861e91f5eba18"}
Sep 08 00:04:02.702829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.702451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq" event={"ID":"060c7c6e-871d-4a3f-b359-0e97a3c12aa0","Type":"ContainerStarted","Data":"a118a4de8a201e32657f8dc4a2b5689b2f2a00bd94b3e4e2ade2f61ad3bf8c1b"}
Sep 08 00:04:02.702829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.702472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq" event={"ID":"060c7c6e-871d-4a3f-b359-0e97a3c12aa0","Type":"ContainerStarted","Data":"9243718fc633fffb93bc82f382804f2aba0d190490b7f436cdb949914de96bec"}
Sep 08 00:04:02.826704 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.826623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"
Sep 08 00:04:02.834669 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:02.834513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ld88q"
Sep 08 00:04:02.843666 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:04:02.843641    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe8f8b5_d294_47ae_8592_40ec7f2b6989.slice/crio-d1a9962687d505d7909583eaa3000850427fd5b57b07ce742295bd57d3620fb6 WatchSource:0}: Error finding container d1a9962687d505d7909583eaa3000850427fd5b57b07ce742295bd57d3620fb6: Status 404 returned error can't find the container with id d1a9962687d505d7909583eaa3000850427fd5b57b07ce742295bd57d3620fb6
Sep 08 00:04:03.095037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:03.094973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg"]
Sep 08 00:04:03.097746 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:04:03.097723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69cd725_f8e7_42c0_8938_9c2a1a119e64.slice/crio-b0c0be5ef6c639dbe274da8c14f652045e7265bd6dfcc8a4e209a96963019d74 WatchSource:0}: Error finding container b0c0be5ef6c639dbe274da8c14f652045e7265bd6dfcc8a4e209a96963019d74: Status 404 returned error can't find the container with id b0c0be5ef6c639dbe274da8c14f652045e7265bd6dfcc8a4e209a96963019d74
Sep 08 00:04:03.706976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:03.706936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld88q" event={"ID":"dfe8f8b5-d294-47ae-8592-40ec7f2b6989","Type":"ContainerStarted","Data":"d1a9962687d505d7909583eaa3000850427fd5b57b07ce742295bd57d3620fb6"}
Sep 08 00:04:03.708369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:03.708343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg" event={"ID":"e69cd725-f8e7-42c0-8938-9c2a1a119e64","Type":"ContainerStarted","Data":"b0c0be5ef6c639dbe274da8c14f652045e7265bd6dfcc8a4e209a96963019d74"}
Sep 08 00:04:04.714868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:04.714835    2580 generic.go:356] "Generic (PLEG): container finished" podID="dfe8f8b5-d294-47ae-8592-40ec7f2b6989" containerID="967f90a17d98cbffe30879594d8ee901b5bab50495b449255a46024f134d5fb8" exitCode=0
Sep 08 00:04:04.715240 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:04.714926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld88q" event={"ID":"dfe8f8b5-d294-47ae-8592-40ec7f2b6989","Type":"ContainerDied","Data":"967f90a17d98cbffe30879594d8ee901b5bab50495b449255a46024f134d5fb8"}
Sep 08 00:04:04.717372 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:04.717344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq" event={"ID":"060c7c6e-871d-4a3f-b359-0e97a3c12aa0","Type":"ContainerStarted","Data":"5f43b96bf3026300a2b21051bf5aa1ec08ea4bde1d369200db6b455b329465ed"}
Sep 08 00:04:04.746183 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:04.746136    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kscxq" podStartSLOduration=2.462002911 podStartE2EDuration="3.746117721s" totalImagesPullingTime="1.28411481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:04.744954355 +0000 UTC m=+76.917219239" watchObservedRunningTime="2026-09-08 00:04:04.746117721 +0000 UTC m=+76.918382604"
Sep 08 00:04:05.723389 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.723346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg" event={"ID":"e69cd725-f8e7-42c0-8938-9c2a1a119e64","Type":"ContainerStarted","Data":"adfed7087f9e58798292e81b24481965d2a80ba88486eef478009957dd222243"}
Sep 08 00:04:05.723847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.723418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg" event={"ID":"e69cd725-f8e7-42c0-8938-9c2a1a119e64","Type":"ContainerStarted","Data":"d0d8bccd94130d2ea9b239da216f65d70bbe7ddeeb72c21f3dcb0a59f8aeaf31"}
Sep 08 00:04:05.723847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.723449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg" event={"ID":"e69cd725-f8e7-42c0-8938-9c2a1a119e64","Type":"ContainerStarted","Data":"6d8b24d5735e7859d55ed962ce933261f6c12057481f705a4d3ff5b4429b886f"}
Sep 08 00:04:05.725512 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.725488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld88q" event={"ID":"dfe8f8b5-d294-47ae-8592-40ec7f2b6989","Type":"ContainerStarted","Data":"ea1b9dcc79b1477f0aa92b349bd7aea5378248c5520aaf68a98ce6d6e149fc47"}
Sep 08 00:04:05.725634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.725553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld88q" event={"ID":"dfe8f8b5-d294-47ae-8592-40ec7f2b6989","Type":"ContainerStarted","Data":"04d16a4603dcc89f3958c1ab703df4e540b12b298ea00f3be6d1ee6878c408a8"}
Sep 08 00:04:05.739418 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.739374    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-dpdvg" podStartSLOduration=3.186186667 podStartE2EDuration="4.739359004s" totalImagesPullingTime="1.553172337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:05.737718095 +0000 UTC m=+77.909982978" watchObservedRunningTime="2026-09-08 00:04:05.739359004 +0000 UTC m=+77.911623887"
Sep 08 00:04:05.754825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:05.754774    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ld88q" podStartSLOduration=3.6646034690000002 podStartE2EDuration="4.75475784s" totalImagesPullingTime="1.025047488s" totalInitContainerRuntime="65.106883ms" isStatefulPod=true podCreationTimestamp="2026-09-08 00:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:05.75423931 +0000 UTC m=+77.926504189" watchObservedRunningTime="2026-09-08 00:04:05.75475784 +0000 UTC m=+77.927022725"
Sep 08 00:04:06.679246 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.679207    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"]
Sep 08 00:04:06.682219 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.682200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.684392 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.684367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dqh6x\""
Sep 08 00:04:06.684392 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.684372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 00:04:06.693076 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.693042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"]
Sep 08 00:04:06.822098 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.822064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b888ecc2-7649-4da3-9272-cdffeca466e3-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.822098 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.822093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkp2v\" (UniqueName: \"kubernetes.io/projected/b888ecc2-7649-4da3-9272-cdffeca466e3-kube-api-access-mkp2v\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.922878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.922838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b888ecc2-7649-4da3-9272-cdffeca466e3-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.922878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.922876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp2v\" (UniqueName: \"kubernetes.io/projected/b888ecc2-7649-4da3-9272-cdffeca466e3-kube-api-access-mkp2v\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.925211 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.925190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b888ecc2-7649-4da3-9272-cdffeca466e3-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.930121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.930049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp2v\" (UniqueName: \"kubernetes.io/projected/b888ecc2-7649-4da3-9272-cdffeca466e3-kube-api-access-mkp2v\") pod \"monitoring-plugin-85bbbbbf4-2th4n\" (UID: \"b888ecc2-7649-4da3-9272-cdffeca466e3\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:06.991553 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:06.991499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:07.256383 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:07.256300    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"]
Sep 08 00:04:07.259049 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:04:07.259021    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb888ecc2_7649_4da3_9272_cdffeca466e3.slice/crio-e8bb71d53141d34454e329fe298c85f92fb57995f5dbee415e7f2faf2b7eda40 WatchSource:0}: Error finding container e8bb71d53141d34454e329fe298c85f92fb57995f5dbee415e7f2faf2b7eda40: Status 404 returned error can't find the container with id e8bb71d53141d34454e329fe298c85f92fb57995f5dbee415e7f2faf2b7eda40
Sep 08 00:04:07.735246 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:07.735212    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n" event={"ID":"b888ecc2-7649-4da3-9272-cdffeca466e3","Type":"ContainerStarted","Data":"e8bb71d53141d34454e329fe298c85f92fb57995f5dbee415e7f2faf2b7eda40"}
Sep 08 00:04:08.122846 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.122751    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:04:08.126798 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.126765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.129072 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.129048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 00:04:08.129477 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.129458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 00:04:08.129605 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.129501    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 00:04:08.129605 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.129565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 00:04:08.129700 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.129613    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 00:04:08.130355 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130206    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 00:04:08.130355 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 00:04:08.130355 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130281    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 00:04:08.130595 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130508    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 00:04:08.130755 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 00:04:08.130841 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130814    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-86qjrm7r4mls6\""
Sep 08 00:04:08.130920 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130893    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 00:04:08.131121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.130999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 00:04:08.131121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.131043    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 00:04:08.131282 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.131227    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 00:04:08.131549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.131514    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g6rbz\""
Sep 08 00:04:08.133184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.133154    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 00:04:08.141377 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.141354    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:04:08.232163 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232163 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjfwt\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.232977 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.232869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334296 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334296 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjfwt\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334434    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.334506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.334971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.335416 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.338410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.335501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.338410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.336142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.338410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.337878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.339401 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.339376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.339502 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.339483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.339609 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.339585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340056 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340165 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340505 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340778 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340778 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.340933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.340843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.341021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.341000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.341987 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.341967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.342219 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.342193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.342219 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.342215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.342787 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.342764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.343605 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.343586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjfwt\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt\") pod \"prometheus-k8s-0\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.438981 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.438950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:08.912339 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:08.912303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:04:08.915431 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:04:08.915397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa68b4fa_8dbd_41a3_9af0_30048417626e.slice/crio-46c97f4af24768610d825c7506d0f8902550236f83aed6cd0a0667210220d950 WatchSource:0}: Error finding container 46c97f4af24768610d825c7506d0f8902550236f83aed6cd0a0667210220d950: Status 404 returned error can't find the container with id 46c97f4af24768610d825c7506d0f8902550236f83aed6cd0a0667210220d950
Sep 08 00:04:09.645505 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.645472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-665fcd494-75bck"
Sep 08 00:04:09.745976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.745874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"46c97f4af24768610d825c7506d0f8902550236f83aed6cd0a0667210220d950"}
Sep 08 00:04:09.747790 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.747761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n" event={"ID":"b888ecc2-7649-4da3-9272-cdffeca466e3","Type":"ContainerStarted","Data":"31abfdc178a5a16b59e091b5d4a7044fecf301b0e75ebc7904c6697f7b846457"}
Sep 08 00:04:09.748015 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.747974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:09.753178 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.753155    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n"
Sep 08 00:04:09.763555 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:09.763495    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-2th4n" podStartSLOduration=2.329502788 podStartE2EDuration="3.763478487s" totalImagesPullingTime="1.433975699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:09.762022014 +0000 UTC m=+81.934286900" watchObservedRunningTime="2026-09-08 00:04:09.763478487 +0000 UTC m=+81.935743383"
Sep 08 00:04:10.753719 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:10.753685    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742" exitCode=0
Sep 08 00:04:10.754062 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:10.753769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742"}
Sep 08 00:04:15.775790 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:15.775553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"}
Sep 08 00:04:15.775790 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:15.775679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"}
Sep 08 00:04:17.786681 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:17.786605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"}
Sep 08 00:04:17.787045 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:17.786698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"}
Sep 08 00:04:17.787045 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:17.786726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"}
Sep 08 00:04:17.787045 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:17.786752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerStarted","Data":"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"}
Sep 08 00:04:17.813754 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:17.813693    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.12258185 podStartE2EDuration="9.813668585s" totalImagesPullingTime="7.673161487s" totalInitContainerRuntime="17.925248ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:04:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:04:17.811745637 +0000 UTC m=+89.984010535" watchObservedRunningTime="2026-09-08 00:04:17.813668585 +0000 UTC m=+89.985933474"
Sep 08 00:04:18.439284 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:18.439238    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:04:32.705574 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:04:32.705524    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rf9cf"
Sep 08 00:05:08.439785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:08.439746    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:08.458807 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:08.458779    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:08.992427 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:08.992395    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:26.503765 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.503729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504076    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="prometheus" containerID="cri-o://52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2" gracePeriod=600
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504106    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="config-reloader" containerID="cri-o://0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30" gracePeriod=600
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504106    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="thanos-sidecar" containerID="cri-o://28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888" gracePeriod=600
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504100    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy" containerID="cri-o://3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b" gracePeriod=600
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504149    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-web" containerID="cri-o://bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8" gracePeriod=600
Sep 08 00:05:26.504247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:26.504200    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4" gracePeriod=600
Sep 08 00:05:27.048260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048229    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4" exitCode=0
Sep 08 00:05:27.048260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048255    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b" exitCode=0
Sep 08 00:05:27.048260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048261    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888" exitCode=0
Sep 08 00:05:27.048260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048268    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30" exitCode=0
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048275    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2" exitCode=0
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"}
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"}
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"}
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"}
Sep 08 00:05:27.048548 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.048436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"}
Sep 08 00:05:27.743762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.743738    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:27.861777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.861692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.861777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.861746    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.861777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.861769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862000 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.861796    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862064 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.861993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862064 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862075    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjfwt\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862365    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out\") pod \"aa68b4fa-8dbd-41a3-9af0-30048417626e\" (UID: \"aa68b4fa-8dbd-41a3-9af0-30048417626e\") "
Sep 08 00:05:27.862953 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.862812    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-metrics-client-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.864906 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.864631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.864906 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.864688    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.864906 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.864707    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.864906 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.864759    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.864906 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.864779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.865229 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865157    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.865714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.865714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.865875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt" (OuterVolumeSpecName: "kube-api-access-hjfwt") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "kube-api-access-hjfwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:05:27.865875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:05:27.865978 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.865904    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.866305 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.866184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out" (OuterVolumeSpecName: "config-out") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 00:05:27.866305 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.866190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.866305 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.866201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.866487 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.866470    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:05:27.866933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.866911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:05:27.867071 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.867046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.867722 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.867700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config" (OuterVolumeSpecName: "config") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.877507 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.877484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config" (OuterVolumeSpecName: "web-config") pod "aa68b4fa-8dbd-41a3-9af0-30048417626e" (UID: "aa68b4fa-8dbd-41a3-9af0-30048417626e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:05:27.963544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963486    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963518    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-config\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963548    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963557    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-config-out\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963568    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963578    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963586    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-kube-rbac-proxy\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963595    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963605    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963613    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjfwt\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-kube-api-access-hjfwt\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963622    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa68b4fa-8dbd-41a3-9af0-30048417626e-tls-assets\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963631    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-web-config\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963638    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963647    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-db\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963654    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-grpc-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963662    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa68b4fa-8dbd-41a3-9af0-30048417626e-secret-metrics-client-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963669    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963678    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:27.963771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:27.963687    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa68b4fa-8dbd-41a3-9af0-30048417626e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:05:28.060180 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.060141    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerID="bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8" exitCode=0
Sep 08 00:05:28.060363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.060192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"}
Sep 08 00:05:28.060363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.060229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.060363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.060289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa68b4fa-8dbd-41a3-9af0-30048417626e","Type":"ContainerDied","Data":"46c97f4af24768610d825c7506d0f8902550236f83aed6cd0a0667210220d950"}
Sep 08 00:05:28.060363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.060334    2580 scope.go:117] "RemoveContainer" containerID="0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"
Sep 08 00:05:28.068766 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.068677    2580 scope.go:117] "RemoveContainer" containerID="3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"
Sep 08 00:05:28.075513 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.075493    2580 scope.go:117] "RemoveContainer" containerID="bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"
Sep 08 00:05:28.084227 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.084197    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:28.085385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.085367    2580 scope.go:117] "RemoveContainer" containerID="28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"
Sep 08 00:05:28.086918 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.086900    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:28.092574 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.092558    2580 scope.go:117] "RemoveContainer" containerID="0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"
Sep 08 00:05:28.099153 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.099137    2580 scope.go:117] "RemoveContainer" containerID="52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"
Sep 08 00:05:28.106282 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.106258    2580 scope.go:117] "RemoveContainer" containerID="9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742"
Sep 08 00:05:28.110181 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110160    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:28.110447 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110430    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="prometheus"
Sep 08 00:05:28.110447 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110446    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="prometheus"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110458    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="thanos-sidecar"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110464    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="thanos-sidecar"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110473    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-web"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110479    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-web"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110489    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110494    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110499    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="init-config-reloader"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110505    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="init-config-reloader"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110510    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-thanos"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110516    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-thanos"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110524    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="config-reloader"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="config-reloader"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110595    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="thanos-sidecar"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110602    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="config-reloader"
Sep 08 00:05:28.110600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110607    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-web"
Sep 08 00:05:28.111068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110613    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="prometheus"
Sep 08 00:05:28.111068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110621    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy"
Sep 08 00:05:28.111068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.110627    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" containerName="kube-rbac-proxy-thanos"
Sep 08 00:05:28.113335 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.113317    2580 scope.go:117] "RemoveContainer" containerID="0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"
Sep 08 00:05:28.113638 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.113614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4\": container with ID starting with 0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4 not found: ID does not exist" containerID="0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"
Sep 08 00:05:28.113714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.113651    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4"} err="failed to get container status \"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4\": rpc error: code = NotFound desc = could not find container \"0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4\": container with ID starting with 0ae7418023513e6a63ffbec3802b82d1fbb6a8a53eaa08899750dabbecdf31d4 not found: ID does not exist"
Sep 08 00:05:28.113714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.113696    2580 scope.go:117] "RemoveContainer" containerID="3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"
Sep 08 00:05:28.113977 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.113957    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b\": container with ID starting with 3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b not found: ID does not exist" containerID="3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"
Sep 08 00:05:28.114016 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.113985    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b"} err="failed to get container status \"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b\": rpc error: code = NotFound desc = could not find container \"3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b\": container with ID starting with 3f671a081bbfa0c422538fa3c685d271a0b573cccbd1fd54d44cd21a84027a9b not found: ID does not exist"
Sep 08 00:05:28.114016 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.114002    2580 scope.go:117] "RemoveContainer" containerID="bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"
Sep 08 00:05:28.114257 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.114241    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8\": container with ID starting with bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8 not found: ID does not exist" containerID="bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"
Sep 08 00:05:28.114316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.114264    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8"} err="failed to get container status \"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8\": rpc error: code = NotFound desc = could not find container \"bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8\": container with ID starting with bfecef66eb9c22f75dc93b231afe2a029af9a66e64e34637795c8d55e454cdd8 not found: ID does not exist"
Sep 08 00:05:28.114316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.114285    2580 scope.go:117] "RemoveContainer" containerID="28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"
Sep 08 00:05:28.114539 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.114509    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888\": container with ID starting with 28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888 not found: ID does not exist" containerID="28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"
Sep 08 00:05:28.114649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.114632    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888"} err="failed to get container status \"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888\": rpc error: code = NotFound desc = could not find container \"28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888\": container with ID starting with 28042a5d188726ebd942e9a1f6d6bf33d316877010497f0d0b3c1443a2f11888 not found: ID does not exist"
Sep 08 00:05:28.114694 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.114651    2580 scope.go:117] "RemoveContainer" containerID="0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"
Sep 08 00:05:28.115295 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.115335 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.115301    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30\": container with ID starting with 0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30 not found: ID does not exist" containerID="0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"
Sep 08 00:05:28.115335 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30"} err="failed to get container status \"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30\": rpc error: code = NotFound desc = could not find container \"0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30\": container with ID starting with 0dec6f1f25a491e50f90f83c1342481b3f28740960c6109f5ab18f6177e1ea30 not found: ID does not exist"
Sep 08 00:05:28.115400 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115336    2580 scope.go:117] "RemoveContainer" containerID="52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"
Sep 08 00:05:28.115665 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.115638    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2\": container with ID starting with 52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2 not found: ID does not exist" containerID="52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"
Sep 08 00:05:28.115746 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2"} err="failed to get container status \"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2\": rpc error: code = NotFound desc = could not find container \"52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2\": container with ID starting with 52cfa6c1ce75d45c7aac63e470d79e27f66e73694b0281e293eb7bd1a132d7c2 not found: ID does not exist"
Sep 08 00:05:28.115746 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115693    2580 scope.go:117] "RemoveContainer" containerID="9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742"
Sep 08 00:05:28.115990 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:05:28.115972    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742\": container with ID starting with 9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742 not found: ID does not exist" containerID="9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742"
Sep 08 00:05:28.116061 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.115997    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742"} err="failed to get container status \"9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742\": rpc error: code = NotFound desc = could not find container \"9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742\": container with ID starting with 9b1584ab754123fbdfa03d517e65e94600d342f431891b60fd78335f7d908742 not found: ID does not exist"
Sep 08 00:05:28.117502 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.117486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 00:05:28.117880 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.117860    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 00:05:28.118138 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 00:05:28.118575 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 00:05:28.118575 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118566    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 00:05:28.118778 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118758    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 00:05:28.118967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118832    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 00:05:28.118967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118762    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g6rbz\""
Sep 08 00:05:28.118967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 00:05:28.118967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.118910    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 00:05:28.119210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.119003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 00:05:28.119210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.119135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 00:05:28.119311 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.119253    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-86qjrm7r4mls6\""
Sep 08 00:05:28.119713 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.119698    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 00:05:28.120601 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.120585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 00:05:28.121226 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.121208    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 00:05:28.124084 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.124067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 00:05:28.128232 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.128211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:28.265816 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.265816 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265816    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ssvg\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-kube-api-access-7ssvg\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.265991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266188    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266283    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.266510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.266350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.329085 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.329051    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa68b4fa-8dbd-41a3-9af0-30048417626e" path="/var/lib/kubelet/pods/aa68b4fa-8dbd-41a3-9af0-30048417626e/volumes"
Sep 08 00:05:28.367712 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367712 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367712 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ssvg\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-kube-api-access-7ssvg\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.367860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.367971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368515 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368515 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368515 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.368717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.368518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.370777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.370749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371007 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.370975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371058    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371686 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371859 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371924 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.371924 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.371881    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.372149 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.372126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.372370 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.372349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.372428 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.372417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.372493 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.372476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e64ffbf-542b-4ba3-863c-34f04da30d57-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.373117 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.373095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-config\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.374627 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.374605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.374958 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.374943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.374998 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.374950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ssvg\" (UniqueName: \"kubernetes.io/projected/0e64ffbf-542b-4ba3-863c-34f04da30d57-kube-api-access-7ssvg\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.375044 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.375031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e64ffbf-542b-4ba3-863c-34f04da30d57-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e64ffbf-542b-4ba3-863c-34f04da30d57\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.424586 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.424548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:05:28.698701 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:28.698677    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 00:05:28.700189 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:05:28.700161    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e64ffbf_542b_4ba3_863c_34f04da30d57.slice/crio-c5234e6e39b64cbec3b533a2dd9a71e850f0f6ca1cfc6214436f891de7d595ca WatchSource:0}: Error finding container c5234e6e39b64cbec3b533a2dd9a71e850f0f6ca1cfc6214436f891de7d595ca: Status 404 returned error can't find the container with id c5234e6e39b64cbec3b533a2dd9a71e850f0f6ca1cfc6214436f891de7d595ca
Sep 08 00:05:29.066351 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:29.066266    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e64ffbf-542b-4ba3-863c-34f04da30d57" containerID="039934da75a4a11c9d1197eb577276443d4ec68e3a43d126bdd8ce00810ea3a2" exitCode=0
Sep 08 00:05:29.066716 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:29.066356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerDied","Data":"039934da75a4a11c9d1197eb577276443d4ec68e3a43d126bdd8ce00810ea3a2"}
Sep 08 00:05:29.066716 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:29.066413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"c5234e6e39b64cbec3b533a2dd9a71e850f0f6ca1cfc6214436f891de7d595ca"}
Sep 08 00:05:30.073451 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"9b9ba3428272b097426de9394b96d7abf0ebc89a087da9b44375a68031220c6f"}
Sep 08 00:05:30.073814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"e607522b49aa247fa222001d572de8cb63fdd32874a2d16b43a3f7308c83c92a"}
Sep 08 00:05:30.073814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"4b97216647977b9a0814033bdbbea8643f059dece8d6b08dffceefbefb07309d"}
Sep 08 00:05:30.073814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"3e98cef56a95e9e1cb6ec22337af499e826cf21a8ff5ab2f22d9f34c2fdd13c3"}
Sep 08 00:05:30.073814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073662    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"9af32ae60059d165f28c1776a87112e5018cab63134fa3a06957fe72c0059c8d"}
Sep 08 00:05:30.073814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.073697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e64ffbf-542b-4ba3-863c-34f04da30d57","Type":"ContainerStarted","Data":"1c3fea39b3618c5a5498be5306c0bfe3f7daf2a4fce548a1ccd27df9381013b7"}
Sep 08 00:05:30.100107 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:30.100024    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.079063621 podStartE2EDuration="2.100005367s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.941746ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:05:30.098249807 +0000 UTC m=+162.270514691" watchObservedRunningTime="2026-09-08 00:05:30.100005367 +0000 UTC m=+162.272270252"
Sep 08 00:05:33.425600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:05:33.425520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:06:28.425080 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:06:28.425037    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:06:28.440368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:06:28.440343    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:06:29.302144 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:06:29.302115    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 00:07:48.227779 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:07:48.227752    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 00:09:02.569265 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.569230    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"]
Sep 08 00:09:02.572429 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.572411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.574804 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.574779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 00:09:02.575563 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.575524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 00:09:02.575656 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.575603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 00:09:02.575656 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.575620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-749nj\""
Sep 08 00:09:02.575656 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.575625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 00:09:02.575927 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.575909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 00:09:02.580596 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.580575    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"]
Sep 08 00:09:02.693095 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.693070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.693241 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.693111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e78f5779-087b-4a14-9505-f4f2e616657f-manager-config\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.693241 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.693139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sff82\" (UniqueName: \"kubernetes.io/projected/e78f5779-087b-4a14-9505-f4f2e616657f-kube-api-access-sff82\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.693241 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.693159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-metrics-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.793551 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.793496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e78f5779-087b-4a14-9505-f4f2e616657f-manager-config\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.793717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.793564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sff82\" (UniqueName: \"kubernetes.io/projected/e78f5779-087b-4a14-9505-f4f2e616657f-kube-api-access-sff82\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.793717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.793587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-metrics-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.793717 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.793629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.794208 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.794181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e78f5779-087b-4a14-9505-f4f2e616657f-manager-config\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.795983 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.795963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-metrics-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.796081 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.796034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e78f5779-087b-4a14-9505-f4f2e616657f-cert\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.807636 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.807594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sff82\" (UniqueName: \"kubernetes.io/projected/e78f5779-087b-4a14-9505-f4f2e616657f-kube-api-access-sff82\") pod \"lws-controller-manager-c54b7b87b-bbvx5\" (UID: \"e78f5779-087b-4a14-9505-f4f2e616657f\") " pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:02.881755 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:02.881680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:03.147930 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:03.147905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"]
Sep 08 00:09:03.150091 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:09:03.150054    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78f5779_087b_4a14_9505_f4f2e616657f.slice/crio-6f527b60025912ef3a64c0f2ecb94882a758e38a36c76d621beed17d74cc2a4f WatchSource:0}: Error finding container 6f527b60025912ef3a64c0f2ecb94882a758e38a36c76d621beed17d74cc2a4f: Status 404 returned error can't find the container with id 6f527b60025912ef3a64c0f2ecb94882a758e38a36c76d621beed17d74cc2a4f
Sep 08 00:09:03.151967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:03.151951    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:09:03.831131 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:03.831094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5" event={"ID":"e78f5779-087b-4a14-9505-f4f2e616657f","Type":"ContainerStarted","Data":"6f527b60025912ef3a64c0f2ecb94882a758e38a36c76d621beed17d74cc2a4f"}
Sep 08 00:09:05.841263 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:05.841186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5" event={"ID":"e78f5779-087b-4a14-9505-f4f2e616657f","Type":"ContainerStarted","Data":"695712f7df71e325b18ee5867e1db87317290a91919be425572b57f1f4b101ca"}
Sep 08 00:09:05.841621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:05.841286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:05.857124 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:05.857078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5" podStartSLOduration=1.473618117 podStartE2EDuration="3.857060444s" totalImagesPullingTime="2.383442327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:09:05.856136798 +0000 UTC m=+378.028401693" watchObservedRunningTime="2026-09-08 00:09:05.857060444 +0000 UTC m=+378.029325381"
Sep 08 00:09:15.150549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.150505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"]
Sep 08 00:09:15.153932 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.153901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.157443 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.157419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 00:09:15.157584 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.157419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 00:09:15.157755 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.157739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-flzw7\""
Sep 08 00:09:15.157984 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.157966    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 00:09:15.158197 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.158182    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 00:09:15.170860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.170832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"]
Sep 08 00:09:15.297794 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.297758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-manifests\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.297982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.297828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.297982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.297857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-tmp\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.297982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.297875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd99h\" (UniqueName: \"kubernetes.io/projected/dac1dd37-4ffe-48d6-8d56-a12c480975d6-kube-api-access-pd99h\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.297982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.297908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-manifests\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-tmp\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd99h\" (UniqueName: \"kubernetes.io/projected/dac1dd37-4ffe-48d6-8d56-a12c480975d6-kube-api-access-pd99h\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.399714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.399691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-manifests\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.401683 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.401656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac1dd37-4ffe-48d6-8d56-a12c480975d6-tmp\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.401808 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.401790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.401880 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.401865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dac1dd37-4ffe-48d6-8d56-a12c480975d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.406485 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.406457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd99h\" (UniqueName: \"kubernetes.io/projected/dac1dd37-4ffe-48d6-8d56-a12c480975d6-kube-api-access-pd99h\") pod \"opendatahub-operator-controller-manager-57cf67b67f-vpvdc\" (UID: \"dac1dd37-4ffe-48d6-8d56-a12c480975d6\") " pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.464323 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.464276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:15.741001 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.740905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"]
Sep 08 00:09:15.745828 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:09:15.745796    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac1dd37_4ffe_48d6_8d56_a12c480975d6.slice/crio-d1e5418c6c1d7c62bd16e2ad50568152fc164f3d5fba91e2a94f4b84aedda3b8 WatchSource:0}: Error finding container d1e5418c6c1d7c62bd16e2ad50568152fc164f3d5fba91e2a94f4b84aedda3b8: Status 404 returned error can't find the container with id d1e5418c6c1d7c62bd16e2ad50568152fc164f3d5fba91e2a94f4b84aedda3b8
Sep 08 00:09:15.883162 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:15.883118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc" event={"ID":"dac1dd37-4ffe-48d6-8d56-a12c480975d6","Type":"ContainerStarted","Data":"d1e5418c6c1d7c62bd16e2ad50568152fc164f3d5fba91e2a94f4b84aedda3b8"}
Sep 08 00:09:16.847658 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:16.847609    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c54b7b87b-bbvx5"
Sep 08 00:09:18.897430 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:18.897390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc" event={"ID":"dac1dd37-4ffe-48d6-8d56-a12c480975d6","Type":"ContainerStarted","Data":"3e8a12298263cd0ebf7b740dcd36a236a6874a74e638e4dff0d3bcb51580e2e2"}
Sep 08 00:09:20.906135 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:20.906096    2580 generic.go:356] "Generic (PLEG): container finished" podID="dac1dd37-4ffe-48d6-8d56-a12c480975d6" containerID="3e8a12298263cd0ebf7b740dcd36a236a6874a74e638e4dff0d3bcb51580e2e2" exitCode=0
Sep 08 00:09:20.906504 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:20.906148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc" event={"ID":"dac1dd37-4ffe-48d6-8d56-a12c480975d6","Type":"ContainerDied","Data":"3e8a12298263cd0ebf7b740dcd36a236a6874a74e638e4dff0d3bcb51580e2e2"}
Sep 08 00:09:21.911934 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:21.911900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc" event={"ID":"dac1dd37-4ffe-48d6-8d56-a12c480975d6","Type":"ContainerStarted","Data":"fb1c0678bc0e6feff2ef6b54655a0197b8068dd5d81abbc10cdd38726b02203a"}
Sep 08 00:09:21.912342 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:21.912206    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:21.931040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:21.930994    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc" podStartSLOduration=2.331034528 podStartE2EDuration="6.930976137s" totalImagesPullingTime="3.54510918s" totalInitContainerRuntime="1.054832429s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:09:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:09:21.929951628 +0000 UTC m=+394.102216507" watchObservedRunningTime="2026-09-08 00:09:21.930976137 +0000 UTC m=+394.103241018"
Sep 08 00:09:32.918479 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:32.918440    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57cf67b67f-vpvdc"
Sep 08 00:09:45.698203 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.698165    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6f5tf"]
Sep 08 00:09:45.700327 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.700310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:45.702622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.702590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 08 00:09:45.702622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.702590    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nn69s\""
Sep 08 00:09:45.708061 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.708040    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6f5tf"]
Sep 08 00:09:45.717345 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.717325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:45.717432 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.717365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzths\" (UniqueName: \"kubernetes.io/projected/c493d3b5-e6cf-45de-bc4f-822c202f5e64-kube-api-access-xzths\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:45.818602 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.818564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:45.818775 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.818629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzths\" (UniqueName: \"kubernetes.io/projected/c493d3b5-e6cf-45de-bc4f-822c202f5e64-kube-api-access-xzths\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:45.818775 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:09:45.818724    2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 08 00:09:45.818856 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:09:45.818793    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert podName:c493d3b5-e6cf-45de-bc4f-822c202f5e64 nodeName:}" failed. No retries permitted until 2026-09-08 00:09:46.318775554 +0000 UTC m=+418.491040416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert") pod "odh-model-controller-7998c67dc4-6f5tf" (UID: "c493d3b5-e6cf-45de-bc4f-822c202f5e64") : secret "odh-model-controller-webhook-cert" not found
Sep 08 00:09:45.826348 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:45.826320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzths\" (UniqueName: \"kubernetes.io/projected/c493d3b5-e6cf-45de-bc4f-822c202f5e64-kube-api-access-xzths\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:46.201685 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.201653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gvgwg"]
Sep 08 00:09:46.203956 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.203938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.205945 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.205926    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 00:09:46.212477 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.212448    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gvgwg"]
Sep 08 00:09:46.221191 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.221168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.221275 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.221197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpg8v\" (UniqueName: \"kubernetes.io/projected/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-kube-api-access-qpg8v\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.321760 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.321716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.321760 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.321756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpg8v\" (UniqueName: \"kubernetes.io/projected/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-kube-api-access-qpg8v\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.321958 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.321781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:46.321958 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:09:46.321885    2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 08 00:09:46.321958 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:09:46.321957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs podName:9d404bdf-5cb8-40d7-90e6-cc9447cbc08d nodeName:}" failed. No retries permitted until 2026-09-08 00:09:46.821938513 +0000 UTC m=+418.994203376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs") pod "model-serving-api-54d89bd5fc-gvgwg" (UID: "9d404bdf-5cb8-40d7-90e6-cc9447cbc08d") : secret "model-serving-api-tls" not found
Sep 08 00:09:46.324219 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.324196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c493d3b5-e6cf-45de-bc4f-822c202f5e64-cert\") pod \"odh-model-controller-7998c67dc4-6f5tf\" (UID: \"c493d3b5-e6cf-45de-bc4f-822c202f5e64\") " pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:46.329593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.329570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpg8v\" (UniqueName: \"kubernetes.io/projected/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-kube-api-access-qpg8v\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.611152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.611051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:46.831510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.830965    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.834374 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.834345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d404bdf-5cb8-40d7-90e6-cc9447cbc08d-tls-certs\") pod \"model-serving-api-54d89bd5fc-gvgwg\" (UID: \"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d\") " pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:46.878738 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:46.878657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6f5tf"]
Sep 08 00:09:46.882057 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:09:46.882023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493d3b5_e6cf_45de_bc4f_822c202f5e64.slice/crio-ba1fcb0b87555b584032c6fb91d041e1a1c7a3554a2a46b279df57596be20912 WatchSource:0}: Error finding container ba1fcb0b87555b584032c6fb91d041e1a1c7a3554a2a46b279df57596be20912: Status 404 returned error can't find the container with id ba1fcb0b87555b584032c6fb91d041e1a1c7a3554a2a46b279df57596be20912
Sep 08 00:09:47.007550 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:47.007488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf" event={"ID":"c493d3b5-e6cf-45de-bc4f-822c202f5e64","Type":"ContainerStarted","Data":"ba1fcb0b87555b584032c6fb91d041e1a1c7a3554a2a46b279df57596be20912"}
Sep 08 00:09:47.115045 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:47.115013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:47.384507 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:47.384474    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-gvgwg"]
Sep 08 00:09:47.387756 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:09:47.387727    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d404bdf_5cb8_40d7_90e6_cc9447cbc08d.slice/crio-81248e498613c266e4e6415c3e4b66930f3504d288c124177a5c61818053e5e7 WatchSource:0}: Error finding container 81248e498613c266e4e6415c3e4b66930f3504d288c124177a5c61818053e5e7: Status 404 returned error can't find the container with id 81248e498613c266e4e6415c3e4b66930f3504d288c124177a5c61818053e5e7
Sep 08 00:09:48.014012 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:48.013972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg" event={"ID":"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d","Type":"ContainerStarted","Data":"81248e498613c266e4e6415c3e4b66930f3504d288c124177a5c61818053e5e7"}
Sep 08 00:09:51.029638 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.029470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg" event={"ID":"9d404bdf-5cb8-40d7-90e6-cc9447cbc08d","Type":"ContainerStarted","Data":"938ecf63319c01b1477499e020d3703a6b03538776804188ec7e295e5cad7644"}
Sep 08 00:09:51.029638 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.029605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:09:51.031069 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.031037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf" event={"ID":"c493d3b5-e6cf-45de-bc4f-822c202f5e64","Type":"ContainerStarted","Data":"408e7bfe67ac850e9a3c75486714471793c04c86363e40dfa21166f138e0568b"}
Sep 08 00:09:51.031204 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.031187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:09:51.044445 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.044184    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg" podStartSLOduration=1.6876574340000001 podStartE2EDuration="5.044167727s" totalImagesPullingTime="3.356510293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:09:51.043813584 +0000 UTC m=+423.216078469" watchObservedRunningTime="2026-09-08 00:09:51.044167727 +0000 UTC m=+423.216432611"
Sep 08 00:09:51.060291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:51.060232    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf" podStartSLOduration=2.19917145 podStartE2EDuration="6.060213326s" totalImagesPullingTime="3.861041876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:09:51.059903414 +0000 UTC m=+423.232168306" watchObservedRunningTime="2026-09-08 00:09:51.060213326 +0000 UTC m=+423.232478210"
Sep 08 00:09:52.553382 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.553347    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"]
Sep 08 00:09:52.556883 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.556865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.559353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.559327    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 00:09:52.559353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.559344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 00:09:52.559517 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.559336    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 00:09:52.559517 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.559404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7s5pn\""
Sep 08 00:09:52.560190 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.560171    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 00:09:52.560190 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.560183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 00:09:52.560190 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.560172    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 00:09:52.567804 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.567779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"]
Sep 08 00:09:52.686010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.685969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686010 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686214 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmr7\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-kube-api-access-6jmr7\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686214 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff41e0b6-8375-4388-a167-f1ca67ff9d27-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686214 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686214 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.686333 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.686222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787589 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787589 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787589 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmr7\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-kube-api-access-6jmr7\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787753 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff41e0b6-8375-4388-a167-f1ca67ff9d27-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787809 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.787870 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.787814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.788676 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.788649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.790654 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.790583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff41e0b6-8375-4388-a167-f1ca67ff9d27-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.790654 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.790644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.790874 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.790850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.791728 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.791707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.803218 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.803183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.803352 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.803270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmr7\" (UniqueName: \"kubernetes.io/projected/ff41e0b6-8375-4388-a167-f1ca67ff9d27-kube-api-access-6jmr7\") pod \"istiod-openshift-gateway-5677ccb49b-vj962\" (UID: \"ff41e0b6-8375-4388-a167-f1ca67ff9d27\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:52.867234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:52.867146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:53.151363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:53.151336    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"]
Sep 08 00:09:53.152639 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:09:53.152601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff41e0b6_8375_4388_a167_f1ca67ff9d27.slice/crio-55546bf509f07a9610fa59b6bb2555b545dffc1e28ee22b2b1f0ace3e8694483 WatchSource:0}: Error finding container 55546bf509f07a9610fa59b6bb2555b545dffc1e28ee22b2b1f0ace3e8694483: Status 404 returned error can't find the container with id 55546bf509f07a9610fa59b6bb2555b545dffc1e28ee22b2b1f0ace3e8694483
Sep 08 00:09:54.045352 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:54.045314    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962" event={"ID":"ff41e0b6-8375-4388-a167-f1ca67ff9d27","Type":"ContainerStarted","Data":"55546bf509f07a9610fa59b6bb2555b545dffc1e28ee22b2b1f0ace3e8694483"}
Sep 08 00:09:56.708052 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:56.708001    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 00:09:56.708311 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:56.708086    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 08 00:09:57.065685 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:57.065572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962" event={"ID":"ff41e0b6-8375-4388-a167-f1ca67ff9d27","Type":"ContainerStarted","Data":"0e6510f4619373cb1f7742af8dc44fec042f57780c60b594624ad5bb95feb063"}
Sep 08 00:09:57.065855 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:57.065752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:09:57.067767 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:57.067731    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-vj962 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 00:09:57.067902 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:57.067795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962" podUID="ff41e0b6-8375-4388-a167-f1ca67ff9d27" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 00:09:57.086644 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:57.086573    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962" podStartSLOduration=1.533494765 podStartE2EDuration="5.086555659s" totalImagesPullingTime="3.553060894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:09:57.084061323 +0000 UTC m=+429.256326205" watchObservedRunningTime="2026-09-08 00:09:57.086555659 +0000 UTC m=+429.258820542"
Sep 08 00:09:58.070652 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:09:58.070623    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-vj962"
Sep 08 00:10:02.038626 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:10:02.038593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6f5tf"
Sep 08 00:10:02.041391 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:10:02.041366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-gvgwg"
Sep 08 00:13:35.429550 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.429480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:13:35.432677 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.432660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.434824 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.434801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 00:13:35.434932 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.434844    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 00:13:35.435699 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.435679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 00:13:35.435789 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.435711    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fctqx\""
Sep 08 00:13:35.435789 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.435714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 00:13:35.445595 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.445572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:13:35.533965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.533926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.533965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.533968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.534175 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.534105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shtn\" (UniqueName: \"kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.534175 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.534137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.635473 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.635430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shtn\" (UniqueName: \"kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.635689 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.635481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.635689 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:13:35.635625    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 08 00:13:35.635806 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:13:35.635708    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls podName:7b2dc967-8793-482c-820c-316efe11c091 nodeName:}" failed. No retries permitted until 2026-09-08 00:13:36.135683872 +0000 UTC m=+648.307948749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls") pod "maas-api-7b4c57c69-vnq6g" (UID: "7b2dc967-8793-482c-820c-316efe11c091") : secret "maas-api-serving-cert" not found
Sep 08 00:13:35.635867 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.635848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.635918 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.635893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.636086 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:13:35.636058    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 08 00:13:35.636218 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:13:35.636125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs podName:7b2dc967-8793-482c-820c-316efe11c091 nodeName:}" failed. No retries permitted until 2026-09-08 00:13:36.136108928 +0000 UTC m=+648.308373804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs") pod "maas-api-7b4c57c69-vnq6g" (UID: "7b2dc967-8793-482c-820c-316efe11c091") : secret "maas-api-metrics-tls" not found
Sep 08 00:13:35.636819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.636795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:35.646521 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:35.646492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shtn\" (UniqueName: \"kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.140903 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.140865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.140903 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.140903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.143296 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.143267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.143412 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.143275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") pod \"maas-api-7b4c57c69-vnq6g\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") " pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.342409 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.342373    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:36.611141 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.611078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:13:36.613391 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:13:36.613364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2dc967_8793_482c_820c_316efe11c091.slice/crio-e3a0d98577ca2349f32eabdddac1fd7e95186a8e09f9acb486f4b8bf3ee06eaa WatchSource:0}: Error finding container e3a0d98577ca2349f32eabdddac1fd7e95186a8e09f9acb486f4b8bf3ee06eaa: Status 404 returned error can't find the container with id e3a0d98577ca2349f32eabdddac1fd7e95186a8e09f9acb486f4b8bf3ee06eaa
Sep 08 00:13:36.974990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:36.974956    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" event={"ID":"7b2dc967-8793-482c-820c-316efe11c091","Type":"ContainerStarted","Data":"e3a0d98577ca2349f32eabdddac1fd7e95186a8e09f9acb486f4b8bf3ee06eaa"}
Sep 08 00:13:39.995956 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:39.995912    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" event={"ID":"7b2dc967-8793-482c-820c-316efe11c091","Type":"ContainerStarted","Data":"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"}
Sep 08 00:13:39.996442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:39.996018    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:13:40.011095 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:40.011036    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" podStartSLOduration=2.6986684739999998 podStartE2EDuration="5.011015019s" totalImagesPullingTime="2.312346545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:13:40.009624836 +0000 UTC m=+652.181889719" watchObservedRunningTime="2026-09-08 00:13:40.011015019 +0000 UTC m=+652.183279903"
Sep 08 00:13:46.006059 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:13:46.006026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:14:05.475179 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.475145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"]
Sep 08 00:14:05.481982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.481957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.487744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.487125    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2xmvm\""
Sep 08 00:14:05.488498 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.488321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 00:14:05.488498 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.488333    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 00:14:05.488498 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.488333    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 00:14:05.490923 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.490900    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"]
Sep 08 00:14:05.588662 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.588662 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e6b966-aa3d-4286-b285-1dd75f78311c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.588888 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.588888 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.588888 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchxm\" (UniqueName: \"kubernetes.io/projected/b5e6b966-aa3d-4286-b285-1dd75f78311c-kube-api-access-zchxm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.588888 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.588849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689591 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e6b966-aa3d-4286-b285-1dd75f78311c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689591 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689792 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689792 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchxm\" (UniqueName: \"kubernetes.io/projected/b5e6b966-aa3d-4286-b285-1dd75f78311c-kube-api-access-zchxm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689792 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.689953 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.689848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.690046 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.690023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.690174 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.690154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.690231 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.690183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.691948 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.691921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b966-aa3d-4286-b285-1dd75f78311c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.692177 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.692159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5e6b966-aa3d-4286-b285-1dd75f78311c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.696347 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.696323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchxm\" (UniqueName: \"kubernetes.io/projected/b5e6b966-aa3d-4286-b285-1dd75f78311c-kube-api-access-zchxm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp\" (UID: \"b5e6b966-aa3d-4286-b285-1dd75f78311c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:05.793763 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:05.793673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:06.066187 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:06.066106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"]
Sep 08 00:14:06.070169 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:14:06.070142    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e6b966_aa3d_4286_b285_1dd75f78311c.slice/crio-dcdc17fcf76237cc1d0c58c2b1dddfd79c4c63a2611a2348f54cfaaaef15a6fa WatchSource:0}: Error finding container dcdc17fcf76237cc1d0c58c2b1dddfd79c4c63a2611a2348f54cfaaaef15a6fa: Status 404 returned error can't find the container with id dcdc17fcf76237cc1d0c58c2b1dddfd79c4c63a2611a2348f54cfaaaef15a6fa
Sep 08 00:14:06.071381 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:06.071362    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:14:06.107402 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:06.107368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp" event={"ID":"b5e6b966-aa3d-4286-b285-1dd75f78311c","Type":"ContainerStarted","Data":"dcdc17fcf76237cc1d0c58c2b1dddfd79c4c63a2611a2348f54cfaaaef15a6fa"}
Sep 08 00:14:07.113273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:07.113237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp" event={"ID":"b5e6b966-aa3d-4286-b285-1dd75f78311c","Type":"ContainerStarted","Data":"e9439c583b10933614a725207ecb46d7b46101d85743aae6ece0d584af80ec11"}
Sep 08 00:14:07.130297 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:07.130245    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp" podStartSLOduration=1.19738075 podStartE2EDuration="2.130223753s" totalImagesPullingTime="932.843003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:14:07.129447525 +0000 UTC m=+679.301712410" watchObservedRunningTime="2026-09-08 00:14:07.130223753 +0000 UTC m=+679.302488638"
Sep 08 00:14:15.794210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:15.794169    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:15.794693 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:15.794248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:15.807055 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:15.807029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:14:16.165495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:14:16.165460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp"
Sep 08 00:15:13.771639 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.771603    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"]
Sep 08 00:15:13.774842 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.774824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.776969 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.776946    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 00:15:13.784828 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.784797    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"]
Sep 08 00:15:13.896192 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896156    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45612d10-3c98-4ba8-958b-7906fa8fb089-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.896375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.896375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.896375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7cgn\" (UniqueName: \"kubernetes.io/projected/45612d10-3c98-4ba8-958b-7906fa8fb089-kube-api-access-n7cgn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.896375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.896558 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.896416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.997795 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45612d10-3c98-4ba8-958b-7906fa8fb089-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7cgn\" (UniqueName: \"kubernetes.io/projected/45612d10-3c98-4ba8-958b-7906fa8fb089-kube-api-access-n7cgn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.997957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.998201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.998222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:13.998319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:13.998287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:14.000210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.000184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45612d10-3c98-4ba8-958b-7906fa8fb089-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:14.000358 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.000338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45612d10-3c98-4ba8-958b-7906fa8fb089-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:14.004988 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.004960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7cgn\" (UniqueName: \"kubernetes.io/projected/45612d10-3c98-4ba8-958b-7906fa8fb089-kube-api-access-n7cgn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6llh9\" (UID: \"45612d10-3c98-4ba8-958b-7906fa8fb089\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:14.087618 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.087499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:14.355414 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.355333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"]
Sep 08 00:15:14.358220 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:15:14.358192    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45612d10_3c98_4ba8_958b_7906fa8fb089.slice/crio-cfdbee8610ebdbb7cf0cbfd6c7d41a2921d1e2c5be19d981450f0df05a057deb WatchSource:0}: Error finding container cfdbee8610ebdbb7cf0cbfd6c7d41a2921d1e2c5be19d981450f0df05a057deb: Status 404 returned error can't find the container with id cfdbee8610ebdbb7cf0cbfd6c7d41a2921d1e2c5be19d981450f0df05a057deb
Sep 08 00:15:14.403678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:14.403641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9" event={"ID":"45612d10-3c98-4ba8-958b-7906fa8fb089","Type":"ContainerStarted","Data":"cfdbee8610ebdbb7cf0cbfd6c7d41a2921d1e2c5be19d981450f0df05a057deb"}
Sep 08 00:15:15.409854 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:15.409818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9" event={"ID":"45612d10-3c98-4ba8-958b-7906fa8fb089","Type":"ContainerStarted","Data":"9a0a1faf1d82856fa1bbfdf19ab20723429f77e840b3dbc533bd9acb15d99bdc"}
Sep 08 00:15:15.424080 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:15.423937    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9" podStartSLOduration=2.202232648 podStartE2EDuration="2.423922567s" totalImagesPullingTime="221.689919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:15:15.423559865 +0000 UTC m=+747.595824749" watchObservedRunningTime="2026-09-08 00:15:15.423922567 +0000 UTC m=+747.596187452"
Sep 08 00:15:21.370226 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.370190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"]
Sep 08 00:15:21.373836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.373813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.375789 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.375759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 00:15:21.383949 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.383923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"]
Sep 08 00:15:21.460258 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtd4\" (UniqueName: \"kubernetes.io/projected/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-kube-api-access-2mtd4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.460403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.460403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.460403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.460403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.460403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.460395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561632 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561677    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.561819 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.561796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtd4\" (UniqueName: \"kubernetes.io/projected/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-kube-api-access-2mtd4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.562204 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.562174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.562319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.562220    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.562319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.562246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.563975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.563953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.564149 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.564131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.568901 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.568877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtd4\" (UniqueName: \"kubernetes.io/projected/4c92334e-cbd6-4002-a49f-0fb4167bcb1f-kube-api-access-2mtd4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk\" (UID: \"4c92334e-cbd6-4002-a49f-0fb4167bcb1f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.685436 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.685399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:21.952545 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:21.952493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"]
Sep 08 00:15:21.954950 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:15:21.954924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c92334e_cbd6_4002_a49f_0fb4167bcb1f.slice/crio-d2a73eafe89d1a9353e53ddc921669ec96196114ca179b420494a228d15d4a36 WatchSource:0}: Error finding container d2a73eafe89d1a9353e53ddc921669ec96196114ca179b420494a228d15d4a36: Status 404 returned error can't find the container with id d2a73eafe89d1a9353e53ddc921669ec96196114ca179b420494a228d15d4a36
Sep 08 00:15:22.442419 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:22.442384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk" event={"ID":"4c92334e-cbd6-4002-a49f-0fb4167bcb1f","Type":"ContainerStarted","Data":"c429a8f69d627fd7c211c3d0b61f1400d6d1954798c2ce0f74008ec22a017613"}
Sep 08 00:15:22.442833 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:22.442435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk" event={"ID":"4c92334e-cbd6-4002-a49f-0fb4167bcb1f","Type":"ContainerStarted","Data":"d2a73eafe89d1a9353e53ddc921669ec96196114ca179b420494a228d15d4a36"}
Sep 08 00:15:22.457993 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:22.457947    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk" podStartSLOduration=1.268582822 podStartE2EDuration="1.457930805s" totalImagesPullingTime="189.347983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:15:22.456805145 +0000 UTC m=+754.629070030" watchObservedRunningTime="2026-09-08 00:15:22.457930805 +0000 UTC m=+754.630195689"
Sep 08 00:15:24.087854 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:24.087814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:24.088262 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:24.087878    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:24.100291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:24.100265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:24.460724 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:24.460698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6llh9"
Sep 08 00:15:31.685998 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:31.685942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:31.686572 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:31.686028    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:31.698573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:31.698550    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:15:32.494088 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:15:32.494059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk"
Sep 08 00:17:36.829354 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.829317    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:17:36.832800 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.832777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.835184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.835147    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4374d7-serving-cert\""
Sep 08 00:17:36.843503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.843478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:17:36.878234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.878199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.878413 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.878251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.878413 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.878277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.878413 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.878302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtrk\" (UniqueName: \"kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.979522 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.979486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.979706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.979551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.979706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.979578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtrk\" (UniqueName: \"kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.979706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.979678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.980287 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.980264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.981831 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.981811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.981943 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.981889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:36.986429 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:36.986406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtrk\" (UniqueName: \"kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk\") pod \"maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:37.164804 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:37.164769    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:37.436793 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:37.436682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:17:37.439452 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:17:37.439422    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0380e69d_0527_4b29_9414_0f5a5322429c.slice/crio-dece0f616e4d46a1dc2beaa19d5c35eb126d61fb550533d473ca6708d4b26a90 WatchSource:0}: Error finding container dece0f616e4d46a1dc2beaa19d5c35eb126d61fb550533d473ca6708d4b26a90: Status 404 returned error can't find the container with id dece0f616e4d46a1dc2beaa19d5c35eb126d61fb550533d473ca6708d4b26a90
Sep 08 00:17:38.029598 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:38.029559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" event={"ID":"0380e69d-0527-4b29-9414-0f5a5322429c","Type":"ContainerStarted","Data":"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"}
Sep 08 00:17:38.029994 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:38.029621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" event={"ID":"0380e69d-0527-4b29-9414-0f5a5322429c","Type":"ContainerStarted","Data":"dece0f616e4d46a1dc2beaa19d5c35eb126d61fb550533d473ca6708d4b26a90"}
Sep 08 00:17:38.029994 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:38.029768    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:38.045319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:38.045271    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" podStartSLOduration=1.576089885 podStartE2EDuration="2.045254219s" totalImagesPullingTime="469.164334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:17:38.043949412 +0000 UTC m=+890.216214300" watchObservedRunningTime="2026-09-08 00:17:38.045254219 +0000 UTC m=+890.217519104"
Sep 08 00:17:43.231821 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.231730    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:17:43.235521 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.235496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.237441 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.237415    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-46c5cdbb-serving-cert\""
Sep 08 00:17:43.246332 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.246295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:17:43.333891 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.333853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt6k\" (UniqueName: \"kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.334066 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.333976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.334066 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.334033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.334159 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.334067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.434975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.434937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt6k\" (UniqueName: \"kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.435432 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.435398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.435549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.435458    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.435619 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.435520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.435619 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:17:43.435559    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-serving-cert: secret "maas-api-e2e-ait-46c5cdbb-serving-cert" not found
Sep 08 00:17:43.435723 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:17:43.435631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls podName:deb34d05-5f32-413a-9ea3-7007b6aa5b6d nodeName:}" failed. No retries permitted until 2026-09-08 00:17:43.93560921 +0000 UTC m=+896.107874078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls") pod "maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" (UID: "deb34d05-5f32-413a-9ea3-7007b6aa5b6d") : secret "maas-api-e2e-ait-46c5cdbb-serving-cert" not found
Sep 08 00:17:43.436294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.436265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.437830 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.437807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.442779 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.442753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt6k\" (UniqueName: \"kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.939898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.939860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:43.942413 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:43.942383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") pod \"maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:44.039843 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:44.039813    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:17:44.147134 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:44.147102    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:44.416511 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:44.416485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:17:44.418761 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:17:44.418732    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb34d05_5f32_413a_9ea3_7007b6aa5b6d.slice/crio-582e1f60cfd9fce977acd3d690cf9875fceaaf01cb5467276e95e23879124ca8 WatchSource:0}: Error finding container 582e1f60cfd9fce977acd3d690cf9875fceaaf01cb5467276e95e23879124ca8: Status 404 returned error can't find the container with id 582e1f60cfd9fce977acd3d690cf9875fceaaf01cb5467276e95e23879124ca8
Sep 08 00:17:45.071573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.071513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" event={"ID":"deb34d05-5f32-413a-9ea3-7007b6aa5b6d","Type":"ContainerStarted","Data":"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"}
Sep 08 00:17:45.071740 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.071636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:45.071740 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.071666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" event={"ID":"deb34d05-5f32-413a-9ea3-7007b6aa5b6d","Type":"ContainerStarted","Data":"582e1f60cfd9fce977acd3d690cf9875fceaaf01cb5467276e95e23879124ca8"}
Sep 08 00:17:45.091233 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.091178    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" podStartSLOduration=1.668601403 podStartE2EDuration="2.091164705s" totalImagesPullingTime="422.563302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:17:45.090418693 +0000 UTC m=+897.262683579" watchObservedRunningTime="2026-09-08 00:17:45.091164705 +0000 UTC m=+897.263429590"
Sep 08 00:17:45.875946 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.875908    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:17:45.879399 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.879374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:45.881694 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.881670    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-90b207-serving-cert\""
Sep 08 00:17:45.888957 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.888928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:17:45.956947 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.956915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:45.957107 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.957070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:45.957182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.957106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:45.957182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:45.957127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2hm\" (UniqueName: \"kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.057964 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.057929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.057964 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.057968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.058205 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.057989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2hm\" (UniqueName: \"kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.058205 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.058015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.058205 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:17:46.058186    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-90b207-serving-cert: secret "maas-api-e2e-autores-90b207-serving-cert" not found
Sep 08 00:17:46.058364 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:17:46.058266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls podName:3cc4c235-735d-4652-861d-e9b153a35f9d nodeName:}" failed. No retries permitted until 2026-09-08 00:17:46.5582433 +0000 UTC m=+898.730508165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls") pod "maas-api-e2e-autores-90b207-6459c686b5-7zn59" (UID: "3cc4c235-735d-4652-861d-e9b153a35f9d") : secret "maas-api-e2e-autores-90b207-serving-cert" not found
Sep 08 00:17:46.058657 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.058637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.060333 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.060314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.068141 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.068119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2hm\" (UniqueName: \"kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.562692 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.562654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.564965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.564939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") pod \"maas-api-e2e-autores-90b207-6459c686b5-7zn59\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:46.792087 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:46.792047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:47.065000 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:47.064971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:17:47.067285 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:17:47.067256    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc4c235_735d_4652_861d_e9b153a35f9d.slice/crio-153aaa195d4864a962e825f7d5451c67c8a83206da6c186bb9e85783620e67c9 WatchSource:0}: Error finding container 153aaa195d4864a962e825f7d5451c67c8a83206da6c186bb9e85783620e67c9: Status 404 returned error can't find the container with id 153aaa195d4864a962e825f7d5451c67c8a83206da6c186bb9e85783620e67c9
Sep 08 00:17:47.082539 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:47.082502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" event={"ID":"3cc4c235-735d-4652-861d-e9b153a35f9d","Type":"ContainerStarted","Data":"153aaa195d4864a962e825f7d5451c67c8a83206da6c186bb9e85783620e67c9"}
Sep 08 00:17:48.089901 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:48.089864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" event={"ID":"3cc4c235-735d-4652-861d-e9b153a35f9d","Type":"ContainerStarted","Data":"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"}
Sep 08 00:17:48.090279 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:48.090003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:17:48.105171 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:48.105117    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" podStartSLOduration=2.741699773 podStartE2EDuration="3.10510263s" totalImagesPullingTime="363.402857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:17:48.103779263 +0000 UTC m=+900.276044148" watchObservedRunningTime="2026-09-08 00:17:48.10510263 +0000 UTC m=+900.277367558"
Sep 08 00:17:51.082578 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:51.082542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:17:54.100815 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:17:54.100782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:18:05.525158 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.525109    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:18:05.530641 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.530616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.532940 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.532916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c4541b-serving-cert\""
Sep 08 00:18:05.537609 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.537582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:18:05.641688 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.641651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9z7\" (UniqueName: \"kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.641688 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.641699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.641897 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.641771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.641897 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.641846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.742812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.742773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.742976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.742885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9z7\" (UniqueName: \"kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.742976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.742908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.742976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.742928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.743078 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:05.743025    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-serving-cert: secret "maas-api-e2e-inf-b-c4541b-serving-cert" not found
Sep 08 00:18:05.743112 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:05.743103    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls podName:c86c20c6-5c48-4812-8c4e-57eeeb8d21bc nodeName:}" failed. No retries permitted until 2026-09-08 00:18:06.243081316 +0000 UTC m=+918.415346192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls") pod "maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" (UID: "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc") : secret "maas-api-e2e-inf-b-c4541b-serving-cert" not found
Sep 08 00:18:05.743740 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.743713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.745442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.745413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:05.752445 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:05.752423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9z7\" (UniqueName: \"kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:06.248327 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:06.248292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:06.250836 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:06.250804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:06.445360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:06.445319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:06.719320 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:06.719291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:18:06.721745 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:18:06.721718    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86c20c6_5c48_4812_8c4e_57eeeb8d21bc.slice/crio-3b6c2713d5be1f23d5bdf353ae1986e7172fd30f515cae14e37bf1bce141b673 WatchSource:0}: Error finding container 3b6c2713d5be1f23d5bdf353ae1986e7172fd30f515cae14e37bf1bce141b673: Status 404 returned error can't find the container with id 3b6c2713d5be1f23d5bdf353ae1986e7172fd30f515cae14e37bf1bce141b673
Sep 08 00:18:07.181147 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:07.181102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" event={"ID":"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc","Type":"ContainerStarted","Data":"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"}
Sep 08 00:18:07.181543 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:07.181172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" event={"ID":"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc","Type":"ContainerStarted","Data":"3b6c2713d5be1f23d5bdf353ae1986e7172fd30f515cae14e37bf1bce141b673"}
Sep 08 00:18:07.181543 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:07.181209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:07.198738 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:07.198688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" podStartSLOduration=1.819379599 podStartE2EDuration="2.198672059s" totalImagesPullingTime="379.29246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:18:07.197040902 +0000 UTC m=+919.369305787" watchObservedRunningTime="2026-09-08 00:18:07.198672059 +0000 UTC m=+919.370936943"
Sep 08 00:18:13.191375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:13.191345    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:18:14.346155 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.346122    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:18:14.346627 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.346387    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" podUID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" containerName="maas-api" containerID="cri-o://8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec" gracePeriod=30
Sep 08 00:18:14.585642 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.585618    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:18:14.725675 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.725633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") pod \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") "
Sep 08 00:18:14.725865 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.725720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs\") pod \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") "
Sep 08 00:18:14.725865 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.725813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dt6k\" (UniqueName: \"kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k\") pod \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") "
Sep 08 00:18:14.725865 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.725854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca\") pod \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\" (UID: \"deb34d05-5f32-413a-9ea3-7007b6aa5b6d\") "
Sep 08 00:18:14.726294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.726251    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "deb34d05-5f32-413a-9ea3-7007b6aa5b6d" (UID: "deb34d05-5f32-413a-9ea3-7007b6aa5b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:18:14.727963 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.727926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k" (OuterVolumeSpecName: "kube-api-access-9dt6k") pod "deb34d05-5f32-413a-9ea3-7007b6aa5b6d" (UID: "deb34d05-5f32-413a-9ea3-7007b6aa5b6d"). InnerVolumeSpecName "kube-api-access-9dt6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:18:14.728038 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.727971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "deb34d05-5f32-413a-9ea3-7007b6aa5b6d" (UID: "deb34d05-5f32-413a-9ea3-7007b6aa5b6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:18:14.728038 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.727987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "deb34d05-5f32-413a-9ea3-7007b6aa5b6d" (UID: "deb34d05-5f32-413a-9ea3-7007b6aa5b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:18:14.826882 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.826842    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:14.826882 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.826875    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt6k\" (UniqueName: \"kubernetes.io/projected/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-kube-api-access-9dt6k\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:14.826882 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.826885    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:14.827111 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:14.826893    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deb34d05-5f32-413a-9ea3-7007b6aa5b6d-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:15.221726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.221692    2580 generic.go:356] "Generic (PLEG): container finished" podID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" containerID="8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec" exitCode=0
Sep 08 00:18:15.221898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.221742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" event={"ID":"deb34d05-5f32-413a-9ea3-7007b6aa5b6d","Type":"ContainerDied","Data":"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"}
Sep 08 00:18:15.221898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.221746    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"
Sep 08 00:18:15.221898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.221795    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz" event={"ID":"deb34d05-5f32-413a-9ea3-7007b6aa5b6d","Type":"ContainerDied","Data":"582e1f60cfd9fce977acd3d690cf9875fceaaf01cb5467276e95e23879124ca8"}
Sep 08 00:18:15.221898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.221829    2580 scope.go:117] "RemoveContainer" containerID="8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"
Sep 08 00:18:15.231154 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.231137    2580 scope.go:117] "RemoveContainer" containerID="8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"
Sep 08 00:18:15.231466 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:15.231448    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec\": container with ID starting with 8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec not found: ID does not exist" containerID="8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"
Sep 08 00:18:15.231571 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.231472    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec"} err="failed to get container status \"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec\": rpc error: code = NotFound desc = could not find container \"8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec\": container with ID starting with 8c5829047c54bdb7e1ef434c21b117174be4179d2e2022f400c91ba7e323beec not found: ID does not exist"
Sep 08 00:18:15.245462 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.245437    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:18:15.250176 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:15.250150    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46c5cdbb-69fc8bbc9-4nknz"]
Sep 08 00:18:16.330540 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:16.330493    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" path="/var/lib/kubelet/pods/deb34d05-5f32-413a-9ea3-7007b6aa5b6d/volumes"
Sep 08 00:18:18.713317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.713281    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:18.713721 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.713657    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" containerName="maas-api"
Sep 08 00:18:18.713721 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.713670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" containerName="maas-api"
Sep 08 00:18:18.713849 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.713830    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb34d05-5f32-413a-9ea3-7007b6aa5b6d" containerName="maas-api"
Sep 08 00:18:18.718138 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.718115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.720480 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.720450    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-693d9ea3-serving-cert\""
Sep 08 00:18:18.726979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.726956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:18.759106 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.759067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.759274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.759121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9t6\" (UniqueName: \"kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.759274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.759171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.759274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.759212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.859924 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.859883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9t6\" (UniqueName: \"kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.860118 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.859932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.860118 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.859995    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.860118 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.860068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.860289 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:18.860160    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-serving-cert: secret "maas-api-e2e-migrate-693d9ea3-serving-cert" not found
Sep 08 00:18:18.860289 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:18.860233    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls podName:c7e029c9-e7b0-41d3-92fb-c1d838572865 nodeName:}" failed. No retries permitted until 2026-09-08 00:18:19.360214666 +0000 UTC m=+931.532479532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls") pod "maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" (UID: "c7e029c9-e7b0-41d3-92fb-c1d838572865") : secret "maas-api-e2e-migrate-693d9ea3-serving-cert" not found
Sep 08 00:18:18.860741 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.860716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.862499 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.862470    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:18.868314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:18.868287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9t6\" (UniqueName: \"kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:19.364431 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:19.364397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:19.366710 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:19.366684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") pod \"maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:19.629798 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:19.629723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:19.897369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:19.897346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:19.899490 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:18:19.899460    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e029c9_e7b0_41d3_92fb_c1d838572865.slice/crio-66d2364fda36f9b81a6581bbe91a5efe02bdbdfdd3247bd79bbbc524ed6f8bcd WatchSource:0}: Error finding container 66d2364fda36f9b81a6581bbe91a5efe02bdbdfdd3247bd79bbbc524ed6f8bcd: Status 404 returned error can't find the container with id 66d2364fda36f9b81a6581bbe91a5efe02bdbdfdd3247bd79bbbc524ed6f8bcd
Sep 08 00:18:20.247264 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:20.247219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" event={"ID":"c7e029c9-e7b0-41d3-92fb-c1d838572865","Type":"ContainerStarted","Data":"66d2364fda36f9b81a6581bbe91a5efe02bdbdfdd3247bd79bbbc524ed6f8bcd"}
Sep 08 00:18:21.253271 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:21.253232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" event={"ID":"c7e029c9-e7b0-41d3-92fb-c1d838572865","Type":"ContainerStarted","Data":"91ebcc0799e4ebd8e75f3facbc29ce6b96277842328b4f18bf0f13dce190f54d"}
Sep 08 00:18:21.253649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:21.253326    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:21.268452 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:21.268379    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" podStartSLOduration=2.531123801 podStartE2EDuration="3.268363847s" totalImagesPullingTime="737.240046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:18:21.266775771 +0000 UTC m=+933.439040667" watchObservedRunningTime="2026-09-08 00:18:21.268363847 +0000 UTC m=+933.440628731"
Sep 08 00:18:27.263265 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:27.263235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:49.228552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.228491    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:49.229102 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.228759    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" podUID="c7e029c9-e7b0-41d3-92fb-c1d838572865" containerName="maas-api" containerID="cri-o://91ebcc0799e4ebd8e75f3facbc29ce6b96277842328b4f18bf0f13dce190f54d" gracePeriod=30
Sep 08 00:18:49.386192 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.386162    2580 generic.go:356] "Generic (PLEG): container finished" podID="c7e029c9-e7b0-41d3-92fb-c1d838572865" containerID="91ebcc0799e4ebd8e75f3facbc29ce6b96277842328b4f18bf0f13dce190f54d" exitCode=0
Sep 08 00:18:49.386359 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.386241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" event={"ID":"c7e029c9-e7b0-41d3-92fb-c1d838572865","Type":"ContainerDied","Data":"91ebcc0799e4ebd8e75f3facbc29ce6b96277842328b4f18bf0f13dce190f54d"}
Sep 08 00:18:49.477022 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.476990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:49.542918 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.542761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca\") pod \"c7e029c9-e7b0-41d3-92fb-c1d838572865\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") "
Sep 08 00:18:49.542918 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.542835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") pod \"c7e029c9-e7b0-41d3-92fb-c1d838572865\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") "
Sep 08 00:18:49.542918 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.542907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj9t6\" (UniqueName: \"kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6\") pod \"c7e029c9-e7b0-41d3-92fb-c1d838572865\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") "
Sep 08 00:18:49.543194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.542939    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs\") pod \"c7e029c9-e7b0-41d3-92fb-c1d838572865\" (UID: \"c7e029c9-e7b0-41d3-92fb-c1d838572865\") "
Sep 08 00:18:49.543194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.543138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7e029c9-e7b0-41d3-92fb-c1d838572865" (UID: "c7e029c9-e7b0-41d3-92fb-c1d838572865"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:18:49.543306 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.543251    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e029c9-e7b0-41d3-92fb-c1d838572865-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:49.547058 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.547020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7e029c9-e7b0-41d3-92fb-c1d838572865" (UID: "c7e029c9-e7b0-41d3-92fb-c1d838572865"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:18:49.549396 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.549371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6" (OuterVolumeSpecName: "kube-api-access-dj9t6") pod "c7e029c9-e7b0-41d3-92fb-c1d838572865" (UID: "c7e029c9-e7b0-41d3-92fb-c1d838572865"). InnerVolumeSpecName "kube-api-access-dj9t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:18:49.549502 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.549399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7e029c9-e7b0-41d3-92fb-c1d838572865" (UID: "c7e029c9-e7b0-41d3-92fb-c1d838572865"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:18:49.643763 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.643726    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj9t6\" (UniqueName: \"kubernetes.io/projected/c7e029c9-e7b0-41d3-92fb-c1d838572865-kube-api-access-dj9t6\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:49.643763 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.643762    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:49.643950 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:49.643772    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7e029c9-e7b0-41d3-92fb-c1d838572865-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:18:50.392603 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:50.392517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv" event={"ID":"c7e029c9-e7b0-41d3-92fb-c1d838572865","Type":"ContainerDied","Data":"66d2364fda36f9b81a6581bbe91a5efe02bdbdfdd3247bd79bbbc524ed6f8bcd"}
Sep 08 00:18:50.393028 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:50.392556    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"
Sep 08 00:18:50.393028 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:50.392667    2580 scope.go:117] "RemoveContainer" containerID="91ebcc0799e4ebd8e75f3facbc29ce6b96277842328b4f18bf0f13dce190f54d"
Sep 08 00:18:50.409673 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:50.409645    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:50.415588 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:50.415565    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-693d9ea3-ddf958b65-ng7lv"]
Sep 08 00:18:52.330758 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:52.330727    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7e029c9-e7b0-41d3-92fb-c1d838572865" path="/var/lib/kubelet/pods/c7e029c9-e7b0-41d3-92fb-c1d838572865/volumes"
Sep 08 00:18:53.944415 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.944377    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:18:53.945513 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.945484    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e029c9-e7b0-41d3-92fb-c1d838572865" containerName="maas-api"
Sep 08 00:18:53.945513 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.945515    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e029c9-e7b0-41d3-92fb-c1d838572865" containerName="maas-api"
Sep 08 00:18:53.945681 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.945673    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e029c9-e7b0-41d3-92fb-c1d838572865" containerName="maas-api"
Sep 08 00:18:53.950282 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.950260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:53.952435 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.952410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-93fccec0-serving-cert\""
Sep 08 00:18:53.957622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:53.957598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:18:54.084647 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.084612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t65c\" (UniqueName: \"kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.084812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.084658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.084812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.084769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.084812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.084788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.185828 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.185788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.185828 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.185830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.186035 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.185877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t65c\" (UniqueName: \"kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.186035 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.185895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.186035 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:54.185988    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-serving-cert: secret "maas-api-e2e-ait-93fccec0-serving-cert" not found
Sep 08 00:18:54.186139 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:18:54.186054    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls podName:01b62d92-5dea-4c9b-b3b6-ecc0ef360c47 nodeName:}" failed. No retries permitted until 2026-09-08 00:18:54.686037846 +0000 UTC m=+966.858302707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls") pod "maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" (UID: "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47") : secret "maas-api-e2e-ait-93fccec0-serving-cert" not found
Sep 08 00:18:54.186587 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.186563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.188254 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.188237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.193845 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.193823    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t65c\" (UniqueName: \"kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.692184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.692148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.694562 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.694510    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") pod \"maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:54.863634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:54.863589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:55.132074 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:55.131994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:18:55.134281 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:18:55.134255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b62d92_5dea_4c9b_b3b6_ecc0ef360c47.slice/crio-eb34133d2974534b50de06e300baac50077e91e98e12984805708731a4999294 WatchSource:0}: Error finding container eb34133d2974534b50de06e300baac50077e91e98e12984805708731a4999294: Status 404 returned error can't find the container with id eb34133d2974534b50de06e300baac50077e91e98e12984805708731a4999294
Sep 08 00:18:55.418409 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:55.418370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" event={"ID":"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47","Type":"ContainerStarted","Data":"eb34133d2974534b50de06e300baac50077e91e98e12984805708731a4999294"}
Sep 08 00:18:56.424907 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:56.424871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" event={"ID":"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47","Type":"ContainerStarted","Data":"74833e177c393ab2ef8744ccaf98c361bba17e322b8a2387fecf59d302020a1b"}
Sep 08 00:18:56.425294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:56.424940    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:18:56.440168 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:18:56.440120    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" podStartSLOduration=3.082156045 podStartE2EDuration="3.440104182s" totalImagesPullingTime="357.948137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:18:56.43934197 +0000 UTC m=+968.611606854" watchObservedRunningTime="2026-09-08 00:18:56.440104182 +0000 UTC m=+968.612369102"
Sep 08 00:19:02.435087 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:02.435055    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:19:24.424638 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.424488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:19:24.425272 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.424785    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" podUID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" containerName="maas-api" containerID="cri-o://74833e177c393ab2ef8744ccaf98c361bba17e322b8a2387fecf59d302020a1b" gracePeriod=30
Sep 08 00:19:24.552519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.552483    2580 generic.go:356] "Generic (PLEG): container finished" podID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" containerID="74833e177c393ab2ef8744ccaf98c361bba17e322b8a2387fecf59d302020a1b" exitCode=0
Sep 08 00:19:24.552672 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.552521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" event={"ID":"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47","Type":"ContainerDied","Data":"74833e177c393ab2ef8744ccaf98c361bba17e322b8a2387fecf59d302020a1b"}
Sep 08 00:19:24.668308 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.668279    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:19:24.764395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.764315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") pod \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") "
Sep 08 00:19:24.764573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.764450    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t65c\" (UniqueName: \"kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c\") pod \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") "
Sep 08 00:19:24.764573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.764471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs\") pod \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") "
Sep 08 00:19:24.764573 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.764509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca\") pod \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\" (UID: \"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47\") "
Sep 08 00:19:24.764905 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.764878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" (UID: "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:19:24.766544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.766505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c" (OuterVolumeSpecName: "kube-api-access-4t65c") pod "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" (UID: "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47"). InnerVolumeSpecName "kube-api-access-4t65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:19:24.766544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.766519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" (UID: "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:19:24.766663 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.766505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" (UID: "01b62d92-5dea-4c9b-b3b6-ecc0ef360c47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:19:24.866068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.866035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t65c\" (UniqueName: \"kubernetes.io/projected/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-kube-api-access-4t65c\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:19:24.866068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.866063    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:19:24.866068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.866072    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:19:24.866295 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:24.866081    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:19:25.557913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:25.557878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh" event={"ID":"01b62d92-5dea-4c9b-b3b6-ecc0ef360c47","Type":"ContainerDied","Data":"eb34133d2974534b50de06e300baac50077e91e98e12984805708731a4999294"}
Sep 08 00:19:25.557913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:25.557898    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"
Sep 08 00:19:25.558360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:25.557938    2580 scope.go:117] "RemoveContainer" containerID="74833e177c393ab2ef8744ccaf98c361bba17e322b8a2387fecf59d302020a1b"
Sep 08 00:19:25.577556 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:25.577513    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:19:25.581210 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:25.581188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-93fccec0-5f4fd4869b-jgqkh"]
Sep 08 00:19:26.331718 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:26.331683    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" path="/var/lib/kubelet/pods/01b62d92-5dea-4c9b-b3b6-ecc0ef360c47/volumes"
Sep 08 00:19:30.120291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.120257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:19:30.120720 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.120700    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" containerName="maas-api"
Sep 08 00:19:30.120798 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.120722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" containerName="maas-api"
Sep 08 00:19:30.120863 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.120829    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b62d92-5dea-4c9b-b3b6-ecc0ef360c47" containerName="maas-api"
Sep 08 00:19:30.125449 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.125424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.127716 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.127683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5de06f3e-serving-cert\""
Sep 08 00:19:30.140460 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.140425    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:19:30.214775 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.214741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jck6\" (UniqueName: \"kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.214775 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.214775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.214996 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.214809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.214996 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.214939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.315638 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.315598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.315839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.315669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.315839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.315728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jck6\" (UniqueName: \"kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.315839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.315745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.315839 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:19:30.315762    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-serving-cert: secret "maas-api-e2e-derive-5de06f3e-serving-cert" not found
Sep 08 00:19:30.316046 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:19:30.315847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls podName:695f415c-3a37-48b0-98d2-5b8d8995e1ce nodeName:}" failed. No retries permitted until 2026-09-08 00:19:30.815829097 +0000 UTC m=+1002.988093958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls") pod "maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" (UID: "695f415c-3a37-48b0-98d2-5b8d8995e1ce") : secret "maas-api-e2e-derive-5de06f3e-serving-cert" not found
Sep 08 00:19:30.316321 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.316304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.318053 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.318032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.323812 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.323791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jck6\" (UniqueName: \"kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.819201 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.819158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.821510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.821478    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") pod \"maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:30.831026 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.831001    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:19:30.834511 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.834492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:30.839345 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.839324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5de06f3e\""
Sep 08 00:19:30.845312 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.845291    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:19:30.920396 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.920369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:30.920561 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:30.920461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmhv\" (UniqueName: \"kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.021081 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.021042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.021446 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.021130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmhv\" (UniqueName: \"kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.021718 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.021696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.028247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.028222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmhv\" (UniqueName: \"kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv\") pod \"payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.036081 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.036061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:31.145518 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.145485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:31.304277 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.304227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:19:31.306495 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:19:31.306468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695f415c_3a37_48b0_98d2_5b8d8995e1ce.slice/crio-eb022242b962b23712cd502d5ccad4e7ceb82fad918c69f1688e04282fb98b4b WatchSource:0}: Error finding container eb022242b962b23712cd502d5ccad4e7ceb82fad918c69f1688e04282fb98b4b: Status 404 returned error can't find the container with id eb022242b962b23712cd502d5ccad4e7ceb82fad918c69f1688e04282fb98b4b
Sep 08 00:19:31.307847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.307832    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:19:31.413439 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.413413    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:19:31.415158 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:19:31.415137    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46919972_fe2a_49d9_b787_0d58832b2cf3.slice/crio-2e7a0304e81b0a37a1df18bd801bfa3ba14186e2dac0c4c906ebfaa524c7487a WatchSource:0}: Error finding container 2e7a0304e81b0a37a1df18bd801bfa3ba14186e2dac0c4c906ebfaa524c7487a: Status 404 returned error can't find the container with id 2e7a0304e81b0a37a1df18bd801bfa3ba14186e2dac0c4c906ebfaa524c7487a
Sep 08 00:19:31.588842 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.588805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" event={"ID":"46919972-fe2a-49d9-b787-0d58832b2cf3","Type":"ContainerStarted","Data":"2e7a0304e81b0a37a1df18bd801bfa3ba14186e2dac0c4c906ebfaa524c7487a"}
Sep 08 00:19:31.589970 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:31.589947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" event={"ID":"695f415c-3a37-48b0-98d2-5b8d8995e1ce","Type":"ContainerStarted","Data":"eb022242b962b23712cd502d5ccad4e7ceb82fad918c69f1688e04282fb98b4b"}
Sep 08 00:19:32.597113 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:32.597071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" event={"ID":"695f415c-3a37-48b0-98d2-5b8d8995e1ce","Type":"ContainerStarted","Data":"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"}
Sep 08 00:19:32.597493 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:32.597197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:32.613015 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:32.612964    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" podStartSLOduration=1.968621218 podStartE2EDuration="2.612943511s" totalImagesPullingTime="644.322293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:19:32.611053516 +0000 UTC m=+1004.783318400" watchObservedRunningTime="2026-09-08 00:19:32.612943511 +0000 UTC m=+1004.785208395"
Sep 08 00:19:34.608743 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:34.608699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" event={"ID":"46919972-fe2a-49d9-b787-0d58832b2cf3","Type":"ContainerStarted","Data":"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"}
Sep 08 00:19:34.609148 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:34.608898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:19:34.623986 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:34.623941    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" podStartSLOduration=2.343772419 podStartE2EDuration="4.623925858s" totalImagesPullingTime="2.280153439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:19:34.622421898 +0000 UTC m=+1006.794686781" watchObservedRunningTime="2026-09-08 00:19:34.623925858 +0000 UTC m=+1006.796190741"
Sep 08 00:19:38.608318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:38.608280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:19:45.616708 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:19:45.616670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:20:00.668709 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.668671    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:20:00.669229 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.668937    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" podUID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" containerName="maas-api" containerID="cri-o://9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89" gracePeriod=30
Sep 08 00:20:00.723862 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.723826    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:20:00.724316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.724086    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" podUID="46919972-fe2a-49d9-b787-0d58832b2cf3" containerName="payload-pre-processing" containerID="cri-o://61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4" gracePeriod=30
Sep 08 00:20:00.973519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.973471    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:20:00.987806 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.987773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca\") pod \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") "
Sep 08 00:20:00.987969 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.987825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") pod \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") "
Sep 08 00:20:00.987969 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.987853    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jck6\" (UniqueName: \"kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6\") pod \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") "
Sep 08 00:20:00.987969 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.987944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs\") pod \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\" (UID: \"695f415c-3a37-48b0-98d2-5b8d8995e1ce\") "
Sep 08 00:20:00.988140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.988107    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "695f415c-3a37-48b0-98d2-5b8d8995e1ce" (UID: "695f415c-3a37-48b0-98d2-5b8d8995e1ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:20:00.988273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.988253    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f415c-3a37-48b0-98d2-5b8d8995e1ce-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:00.991443 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.991415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "695f415c-3a37-48b0-98d2-5b8d8995e1ce" (UID: "695f415c-3a37-48b0-98d2-5b8d8995e1ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:20:00.997583 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.997469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "695f415c-3a37-48b0-98d2-5b8d8995e1ce" (UID: "695f415c-3a37-48b0-98d2-5b8d8995e1ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:20:00.997945 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:00.997911    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6" (OuterVolumeSpecName: "kube-api-access-2jck6") pod "695f415c-3a37-48b0-98d2-5b8d8995e1ce" (UID: "695f415c-3a37-48b0-98d2-5b8d8995e1ce"). InnerVolumeSpecName "kube-api-access-2jck6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:20:01.023745 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.023714    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:20:01.089615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgmhv\" (UniqueName: \"kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv\") pod \"46919972-fe2a-49d9-b787-0d58832b2cf3\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") "
Sep 08 00:20:01.089786 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume\") pod \"46919972-fe2a-49d9-b787-0d58832b2cf3\" (UID: \"46919972-fe2a-49d9-b787-0d58832b2cf3\") "
Sep 08 00:20:01.089921 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089904    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:01.089979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089925    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f415c-3a37-48b0-98d2-5b8d8995e1ce-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:01.089979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089939    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jck6\" (UniqueName: \"kubernetes.io/projected/695f415c-3a37-48b0-98d2-5b8d8995e1ce-kube-api-access-2jck6\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:01.090045 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.089987    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46919972-fe2a-49d9-b787-0d58832b2cf3" (UID: "46919972-fe2a-49d9-b787-0d58832b2cf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:20:01.091756 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.091736    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv" (OuterVolumeSpecName: "kube-api-access-bgmhv") pod "46919972-fe2a-49d9-b787-0d58832b2cf3" (UID: "46919972-fe2a-49d9-b787-0d58832b2cf3"). InnerVolumeSpecName "kube-api-access-bgmhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:20:01.190556 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.190469    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46919972-fe2a-49d9-b787-0d58832b2cf3-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:01.190556 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.190496    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmhv\" (UniqueName: \"kubernetes.io/projected/46919972-fe2a-49d9-b787-0d58832b2cf3-kube-api-access-bgmhv\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:01.738697 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.738662    2580 generic.go:356] "Generic (PLEG): container finished" podID="46919972-fe2a-49d9-b787-0d58832b2cf3" containerID="61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4" exitCode=0
Sep 08 00:20:01.739125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.738712    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"
Sep 08 00:20:01.739125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.738744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" event={"ID":"46919972-fe2a-49d9-b787-0d58832b2cf3","Type":"ContainerDied","Data":"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"}
Sep 08 00:20:01.739125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.738796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g" event={"ID":"46919972-fe2a-49d9-b787-0d58832b2cf3","Type":"ContainerDied","Data":"2e7a0304e81b0a37a1df18bd801bfa3ba14186e2dac0c4c906ebfaa524c7487a"}
Sep 08 00:20:01.739125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.738826    2580 scope.go:117] "RemoveContainer" containerID="61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"
Sep 08 00:20:01.740256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.740127    2580 generic.go:356] "Generic (PLEG): container finished" podID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" containerID="9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89" exitCode=0
Sep 08 00:20:01.740256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.740170    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"
Sep 08 00:20:01.740256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.740227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" event={"ID":"695f415c-3a37-48b0-98d2-5b8d8995e1ce","Type":"ContainerDied","Data":"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"}
Sep 08 00:20:01.740425 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.740281    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg" event={"ID":"695f415c-3a37-48b0-98d2-5b8d8995e1ce","Type":"ContainerDied","Data":"eb022242b962b23712cd502d5ccad4e7ceb82fad918c69f1688e04282fb98b4b"}
Sep 08 00:20:01.748342 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.748316    2580 scope.go:117] "RemoveContainer" containerID="61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"
Sep 08 00:20:01.748653 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:20:01.748628    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4\": container with ID starting with 61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4 not found: ID does not exist" containerID="61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"
Sep 08 00:20:01.748727 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.748667    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4"} err="failed to get container status \"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4\": rpc error: code = NotFound desc = could not find container \"61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4\": container with ID starting with 61ef3d8129c5e4fd3157c3c08b34e67ff1e098f2468e0157c7053bbae49aafc4 not found: ID does not exist"
Sep 08 00:20:01.748727 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.748691    2580 scope.go:117] "RemoveContainer" containerID="9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"
Sep 08 00:20:01.756373 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.756355    2580 scope.go:117] "RemoveContainer" containerID="9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"
Sep 08 00:20:01.756686 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:20:01.756667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89\": container with ID starting with 9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89 not found: ID does not exist" containerID="9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"
Sep 08 00:20:01.756738 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.756692    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89"} err="failed to get container status \"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89\": rpc error: code = NotFound desc = could not find container \"9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89\": container with ID starting with 9f19e49198f255edcc853f379399003ae08f10000ffa46b633c47aaa83154c89 not found: ID does not exist"
Sep 08 00:20:01.764855 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.764835    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:20:01.768777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.768757    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5de06f3e-75dcf5db4b-svp5g"]
Sep 08 00:20:01.780735 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.780714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:20:01.787452 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:01.787434    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5de06f3e-6588ff7b59-hcxlg"]
Sep 08 00:20:02.331345 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:02.331311    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46919972-fe2a-49d9-b787-0d58832b2cf3" path="/var/lib/kubelet/pods/46919972-fe2a-49d9-b787-0d58832b2cf3/volumes"
Sep 08 00:20:02.331724 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:02.331710    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" path="/var/lib/kubelet/pods/695f415c-3a37-48b0-98d2-5b8d8995e1ce/volumes"
Sep 08 00:20:29.760232 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:29.760195    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:20:29.761289 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:29.761234    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" podUID="3cc4c235-735d-4652-861d-e9b153a35f9d" containerName="maas-api" containerID="cri-o://d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91" gracePeriod=30
Sep 08 00:20:30.010500 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.010444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:20:30.150140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.150107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw2hm\" (UniqueName: \"kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm\") pod \"3cc4c235-735d-4652-861d-e9b153a35f9d\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") "
Sep 08 00:20:30.150318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.150154    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca\") pod \"3cc4c235-735d-4652-861d-e9b153a35f9d\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") "
Sep 08 00:20:30.150318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.150192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs\") pod \"3cc4c235-735d-4652-861d-e9b153a35f9d\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") "
Sep 08 00:20:30.150318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.150238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") pod \"3cc4c235-735d-4652-861d-e9b153a35f9d\" (UID: \"3cc4c235-735d-4652-861d-e9b153a35f9d\") "
Sep 08 00:20:30.150582 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.150553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cc4c235-735d-4652-861d-e9b153a35f9d" (UID: "3cc4c235-735d-4652-861d-e9b153a35f9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:20:30.152218 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.152192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3cc4c235-735d-4652-861d-e9b153a35f9d" (UID: "3cc4c235-735d-4652-861d-e9b153a35f9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:20:30.152298 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.152274    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cc4c235-735d-4652-861d-e9b153a35f9d" (UID: "3cc4c235-735d-4652-861d-e9b153a35f9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:20:30.152406 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.152385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm" (OuterVolumeSpecName: "kube-api-access-sw2hm") pod "3cc4c235-735d-4652-861d-e9b153a35f9d" (UID: "3cc4c235-735d-4652-861d-e9b153a35f9d"). InnerVolumeSpecName "kube-api-access-sw2hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:20:30.251785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.251755    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw2hm\" (UniqueName: \"kubernetes.io/projected/3cc4c235-735d-4652-861d-e9b153a35f9d-kube-api-access-sw2hm\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:30.251785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.251783    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4c235-735d-4652-861d-e9b153a35f9d-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:30.251978 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.251796    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:30.251978 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.251811    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cc4c235-735d-4652-861d-e9b153a35f9d-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:20:30.877152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.877116    2580 generic.go:356] "Generic (PLEG): container finished" podID="3cc4c235-735d-4652-861d-e9b153a35f9d" containerID="d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91" exitCode=0
Sep 08 00:20:30.877651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.877155    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"
Sep 08 00:20:30.877651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.877178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" event={"ID":"3cc4c235-735d-4652-861d-e9b153a35f9d","Type":"ContainerDied","Data":"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"}
Sep 08 00:20:30.877651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.877220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59" event={"ID":"3cc4c235-735d-4652-861d-e9b153a35f9d","Type":"ContainerDied","Data":"153aaa195d4864a962e825f7d5451c67c8a83206da6c186bb9e85783620e67c9"}
Sep 08 00:20:30.877651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.877251    2580 scope.go:117] "RemoveContainer" containerID="d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"
Sep 08 00:20:30.885852 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.885821    2580 scope.go:117] "RemoveContainer" containerID="d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"
Sep 08 00:20:30.886097 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:20:30.886077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91\": container with ID starting with d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91 not found: ID does not exist" containerID="d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"
Sep 08 00:20:30.886151 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.886106    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91"} err="failed to get container status \"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91\": rpc error: code = NotFound desc = could not find container \"d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91\": container with ID starting with d040f053711d744797eeef00e7122a6642cf3db4a638bd73b029f382f0c07c91 not found: ID does not exist"
Sep 08 00:20:30.891065 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.891044    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:20:30.896733 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:30.896715    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-90b207-6459c686b5-7zn59"]
Sep 08 00:20:32.329839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:32.329801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc4c235-735d-4652-861d-e9b153a35f9d" path="/var/lib/kubelet/pods/3cc4c235-735d-4652-861d-e9b153a35f9d/volumes"
Sep 08 00:20:41.257314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.257218    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:20:41.257826 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.257805    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc4c235-735d-4652-861d-e9b153a35f9d" containerName="maas-api"
Sep 08 00:20:41.257989 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.257965    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc4c235-735d-4652-861d-e9b153a35f9d" containerName="maas-api"
Sep 08 00:20:41.258121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.257995    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" containerName="maas-api"
Sep 08 00:20:41.258121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258005    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" containerName="maas-api"
Sep 08 00:20:41.258121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258049    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46919972-fe2a-49d9-b787-0d58832b2cf3" containerName="payload-pre-processing"
Sep 08 00:20:41.258121 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46919972-fe2a-49d9-b787-0d58832b2cf3" containerName="payload-pre-processing"
Sep 08 00:20:41.258315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258151    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc4c235-735d-4652-861d-e9b153a35f9d" containerName="maas-api"
Sep 08 00:20:41.258315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258166    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f415c-3a37-48b0-98d2-5b8d8995e1ce" containerName="maas-api"
Sep 08 00:20:41.258315 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.258177    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46919972-fe2a-49d9-b787-0d58832b2cf3" containerName="payload-pre-processing"
Sep 08 00:20:41.262544 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.262508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.265356 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.265339    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-75e439-serving-cert\""
Sep 08 00:20:41.285503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.285476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:20:41.352967 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.352935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.353152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.352977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.353152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.353076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.353152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.353119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.454125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.454089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.454125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.454127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.454319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.454159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.454319 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.454180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.454870 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.454846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.456590 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.456569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.456664 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.456631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.461714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.461695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl\") pod \"maas-api-e2e-explicit-75e439-5fc8758f5-cm275\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.572601 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.572508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:41.840359 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.840329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:20:41.842906 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:20:41.842882    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9aa4328_49d6_4b58_bbab_ea3fb6278510.slice/crio-1eeb92cecffe64972f079174e04230dca7e89afeed746a2cf5c93d4fac627974 WatchSource:0}: Error finding container 1eeb92cecffe64972f079174e04230dca7e89afeed746a2cf5c93d4fac627974: Status 404 returned error can't find the container with id 1eeb92cecffe64972f079174e04230dca7e89afeed746a2cf5c93d4fac627974
Sep 08 00:20:41.929133 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.929097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" event={"ID":"c9aa4328-49d6-4b58-bbab-ea3fb6278510","Type":"ContainerStarted","Data":"1eeb92cecffe64972f079174e04230dca7e89afeed746a2cf5c93d4fac627974"}
Sep 08 00:20:41.991641 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.991602    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:20:41.996326 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.996305    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:41.998503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:41.998476    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-75e439\""
Sep 08 00:20:42.005987 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.005959    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:20:42.029314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.029284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:20:42.032725 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.032709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.034837 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.034817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-75e439-dockercfg-8tspd\""
Sep 08 00:20:42.041417 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.041390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:20:42.059630 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.059598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6njb\" (UniqueName: \"kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.059740 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.059637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.059740 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.059667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.059818 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.059743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64gh\" (UniqueName: \"kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.160193 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.160158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64gh\" (UniqueName: \"kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.160391 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.160209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6njb\" (UniqueName: \"kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.160391 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.160254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.160391 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.160278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.161037 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.161011    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.161126 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.161018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.168929 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.168897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64gh\" (UniqueName: \"kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh\") pod \"payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.169116 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.169094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6njb\" (UniqueName: \"kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb\") pod \"payload-processing-e2e-explicit-75e439-84876c98b5-s4747\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") " pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.308318 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.308280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.344333 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.344299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.583613 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.583587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:20:42.585459 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:20:42.585428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad263fea_b60a_4c20_b876_30c4b8d5560e.slice/crio-a278767df1dd3ac599d4a490066de4ed97affa998b540b86f4655cdabe91ea56 WatchSource:0}: Error finding container a278767df1dd3ac599d4a490066de4ed97affa998b540b86f4655cdabe91ea56: Status 404 returned error can't find the container with id a278767df1dd3ac599d4a490066de4ed97affa998b540b86f4655cdabe91ea56
Sep 08 00:20:42.625346 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.625322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:20:42.628513 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:20:42.628487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc491686_a4c9_4eec_8daf_1c1f4be9a200.slice/crio-0e9c300262dd41ce92b872714cc509399060334304a17e8f3355eb233d8e56aa WatchSource:0}: Error finding container 0e9c300262dd41ce92b872714cc509399060334304a17e8f3355eb233d8e56aa: Status 404 returned error can't find the container with id 0e9c300262dd41ce92b872714cc509399060334304a17e8f3355eb233d8e56aa
Sep 08 00:20:42.935976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.935860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" event={"ID":"c9aa4328-49d6-4b58-bbab-ea3fb6278510","Type":"ContainerStarted","Data":"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"}
Sep 08 00:20:42.935976 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.935930    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:42.937875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.937810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" event={"ID":"ad263fea-b60a-4c20-b876-30c4b8d5560e","Type":"ContainerStarted","Data":"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"}
Sep 08 00:20:42.937875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.937856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" event={"ID":"ad263fea-b60a-4c20-b876-30c4b8d5560e","Type":"ContainerStarted","Data":"a278767df1dd3ac599d4a490066de4ed97affa998b540b86f4655cdabe91ea56"}
Sep 08 00:20:42.938051 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.937887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:20:42.939588 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.939557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" event={"ID":"bc491686-a4c9-4eec-8daf-1c1f4be9a200","Type":"ContainerStarted","Data":"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"}
Sep 08 00:20:42.939708 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.939612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" event={"ID":"bc491686-a4c9-4eec-8daf-1c1f4be9a200","Type":"ContainerStarted","Data":"0e9c300262dd41ce92b872714cc509399060334304a17e8f3355eb233d8e56aa"}
Sep 08 00:20:42.939708 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.939640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:42.951829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.951785    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" podStartSLOduration=1.3842084749999999 podStartE2EDuration="1.951769998s" totalImagesPullingTime="567.561523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:20:42.950607481 +0000 UTC m=+1075.122872365" watchObservedRunningTime="2026-09-08 00:20:42.951769998 +0000 UTC m=+1075.124034882"
Sep 08 00:20:42.968744 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.968691    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" podStartSLOduration=0.968671022 podStartE2EDuration="968.671022ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:20:42.96762387 +0000 UTC m=+1075.139888757" watchObservedRunningTime="2026-09-08 00:20:42.968671022 +0000 UTC m=+1075.140935907"
Sep 08 00:20:42.987651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:42.987608    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" podStartSLOduration=1.9875950310000001 podStartE2EDuration="1.987595031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:20:42.986341877 +0000 UTC m=+1075.158606760" watchObservedRunningTime="2026-09-08 00:20:42.987595031 +0000 UTC m=+1075.159859946"
Sep 08 00:20:48.951231 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:48.951196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:20:53.948422 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:53.948385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:20:53.948911 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:20:53.948468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:21:39.734771 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.734737    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:21:39.738354 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.738331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.741156 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.741131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e250ebf1-serving-cert\""
Sep 08 00:21:39.759894 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.759852    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:21:39.792619 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.792582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllvz\" (UniqueName: \"kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.792805 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.792728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.792805 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.792763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.792805 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.792801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.893403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.893366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.893403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.893406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.893703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.893430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.893703 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:39.893520    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-serving-cert: secret "maas-api-e2e-mt-e250ebf1-serving-cert" not found
Sep 08 00:21:39.893703 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:39.893613    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls podName:ef9eda10-785c-4256-99ab-21ccbb615dc8 nodeName:}" failed. No retries permitted until 2026-09-08 00:21:40.393596011 +0000 UTC m=+1132.565860886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls") pod "maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" (UID: "ef9eda10-785c-4256-99ab-21ccbb615dc8") : secret "maas-api-e2e-mt-e250ebf1-serving-cert" not found
Sep 08 00:21:39.893703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.893626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllvz\" (UniqueName: \"kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.894158 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.894132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.895958 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.895938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:39.903504 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:39.903482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllvz\" (UniqueName: \"kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:40.392513 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.392480    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:21:40.396283 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.396260    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.398519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.398500    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e250ebf1\""
Sep 08 00:21:40.398652 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.398580    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e250ebf1-dockercfg-9rdf4\""
Sep 08 00:21:40.398889 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.398870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:40.401398 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.401375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") pod \"maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:40.407450 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.407426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:21:40.500501 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.500460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.500682 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.500629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nq57\" (UniqueName: \"kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.601341 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.601304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.601503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.601461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nq57\" (UniqueName: \"kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.602126 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.602105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.608876 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.608840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nq57\" (UniqueName: \"kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57\") pod \"payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") " pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.649185 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.649094    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:40.717060 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.717016    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:40.918385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.918351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:21:40.920707 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:21:40.920679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9eda10_785c_4256_99ab_21ccbb615dc8.slice/crio-314013b95ef627d67f32c561a9833b93f954de59f21ed80eb0528623af64c7ff WatchSource:0}: Error finding container 314013b95ef627d67f32c561a9833b93f954de59f21ed80eb0528623af64c7ff: Status 404 returned error can't find the container with id 314013b95ef627d67f32c561a9833b93f954de59f21ed80eb0528623af64c7ff
Sep 08 00:21:40.990735 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:40.990709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:21:40.993295 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:21:40.993265    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7636c0ca_670d_4266_a6fc_b996f7529d85.slice/crio-e87d261675f0dfa7cbd0e0a3fc54ec04a81a5a20924b4a9696589353343ae7e9 WatchSource:0}: Error finding container e87d261675f0dfa7cbd0e0a3fc54ec04a81a5a20924b4a9696589353343ae7e9: Status 404 returned error can't find the container with id e87d261675f0dfa7cbd0e0a3fc54ec04a81a5a20924b4a9696589353343ae7e9
Sep 08 00:21:41.229117 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:41.229028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" event={"ID":"ef9eda10-785c-4256-99ab-21ccbb615dc8","Type":"ContainerStarted","Data":"314013b95ef627d67f32c561a9833b93f954de59f21ed80eb0528623af64c7ff"}
Sep 08 00:21:41.230699 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:41.230667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" event={"ID":"7636c0ca-670d-4266-a6fc-b996f7529d85","Type":"ContainerStarted","Data":"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"}
Sep 08 00:21:41.230840 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:41.230712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" event={"ID":"7636c0ca-670d-4266-a6fc-b996f7529d85","Type":"ContainerStarted","Data":"e87d261675f0dfa7cbd0e0a3fc54ec04a81a5a20924b4a9696589353343ae7e9"}
Sep 08 00:21:41.230840 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:41.230805    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:41.245311 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:41.245260    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" podStartSLOduration=1.245242045 podStartE2EDuration="1.245242045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:21:41.243867057 +0000 UTC m=+1133.416131932" watchObservedRunningTime="2026-09-08 00:21:41.245242045 +0000 UTC m=+1133.417506930"
Sep 08 00:21:42.237495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.237456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" event={"ID":"ef9eda10-785c-4256-99ab-21ccbb615dc8","Type":"ContainerStarted","Data":"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"}
Sep 08 00:21:42.237896 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.237669    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:42.252604 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.252549    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" podStartSLOduration=2.941840961 podStartE2EDuration="3.252511936s" totalImagesPullingTime="310.670975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:21:42.252061653 +0000 UTC m=+1134.424326540" watchObservedRunningTime="2026-09-08 00:21:42.252511936 +0000 UTC m=+1134.424776821"
Sep 08 00:21:42.299231 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.299189    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:21:42.299436 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.299411    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" podUID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" containerName="maas-api" containerID="cri-o://503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c" gracePeriod=30
Sep 08 00:21:42.356583 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.356549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:21:42.356834 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.356805    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" podUID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" containerName="payload-processing" containerID="cri-o://60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f" gracePeriod=30
Sep 08 00:21:42.365384 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.365356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:21:42.365593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.365572    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" podUID="ad263fea-b60a-4c20-b876-30c4b8d5560e" containerName="payload-pre-processing" containerID="cri-o://492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04" gracePeriod=30
Sep 08 00:21:42.622291 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.620924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:21:42.646814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.646785    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:21:42.673232 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.673203    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:21:42.719393 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6njb\" (UniqueName: \"kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb\") pod \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") "
Sep 08 00:21:42.719593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls\") pod \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") "
Sep 08 00:21:42.719593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719472    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca\") pod \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") "
Sep 08 00:21:42.719593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs\") pod \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") "
Sep 08 00:21:42.719593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl\") pod \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\" (UID: \"c9aa4328-49d6-4b58-bbab-ea3fb6278510\") "
Sep 08 00:21:42.719831 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume\") pod \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\" (UID: \"bc491686-a4c9-4eec-8daf-1c1f4be9a200\") "
Sep 08 00:21:42.719892 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.719840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9aa4328-49d6-4b58-bbab-ea3fb6278510" (UID: "c9aa4328-49d6-4b58-bbab-ea3fb6278510"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:21:42.720297 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.720245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc491686-a4c9-4eec-8daf-1c1f4be9a200" (UID: "bc491686-a4c9-4eec-8daf-1c1f4be9a200"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:21:42.721677 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.721647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb" (OuterVolumeSpecName: "kube-api-access-v6njb") pod "bc491686-a4c9-4eec-8daf-1c1f4be9a200" (UID: "bc491686-a4c9-4eec-8daf-1c1f4be9a200"). InnerVolumeSpecName "kube-api-access-v6njb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:21:42.721776 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.721679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl" (OuterVolumeSpecName: "kube-api-access-zq8zl") pod "c9aa4328-49d6-4b58-bbab-ea3fb6278510" (UID: "c9aa4328-49d6-4b58-bbab-ea3fb6278510"). InnerVolumeSpecName "kube-api-access-zq8zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:21:42.721776 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.721683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9aa4328-49d6-4b58-bbab-ea3fb6278510" (UID: "c9aa4328-49d6-4b58-bbab-ea3fb6278510"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:21:42.721855 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.721777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9aa4328-49d6-4b58-bbab-ea3fb6278510" (UID: "c9aa4328-49d6-4b58-bbab-ea3fb6278510"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:21:42.821103 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821009    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume\") pod \"ad263fea-b60a-4c20-b876-30c4b8d5560e\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") "
Sep 08 00:21:42.821103 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64gh\" (UniqueName: \"kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh\") pod \"ad263fea-b60a-4c20-b876-30c4b8d5560e\" (UID: \"ad263fea-b60a-4c20-b876-30c4b8d5560e\") "
Sep 08 00:21:42.821326 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821309    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821330    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9aa4328-49d6-4b58-bbab-ea3fb6278510-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821344    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9aa4328-49d6-4b58-bbab-ea3fb6278510-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821358    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/c9aa4328-49d6-4b58-bbab-ea3fb6278510-kube-api-access-zq8zl\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821385 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821373    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc491686-a4c9-4eec-8daf-1c1f4be9a200-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821516 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821387    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6njb\" (UniqueName: \"kubernetes.io/projected/bc491686-a4c9-4eec-8daf-1c1f4be9a200-kube-api-access-v6njb\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.821516 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.821445    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad263fea-b60a-4c20-b876-30c4b8d5560e" (UID: "ad263fea-b60a-4c20-b876-30c4b8d5560e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:21:42.823105 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.823086    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh" (OuterVolumeSpecName: "kube-api-access-l64gh") pod "ad263fea-b60a-4c20-b876-30c4b8d5560e" (UID: "ad263fea-b60a-4c20-b876-30c4b8d5560e"). InnerVolumeSpecName "kube-api-access-l64gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:21:42.922546 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.922506    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad263fea-b60a-4c20-b876-30c4b8d5560e-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:42.922702 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:42.922554    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64gh\" (UniqueName: \"kubernetes.io/projected/ad263fea-b60a-4c20-b876-30c4b8d5560e-kube-api-access-l64gh\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:43.244693 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.244656    2580 generic.go:356] "Generic (PLEG): container finished" podID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" containerID="60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f" exitCode=0
Sep 08 00:21:43.245142 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.244708    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"
Sep 08 00:21:43.245142 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.244750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" event={"ID":"bc491686-a4c9-4eec-8daf-1c1f4be9a200","Type":"ContainerDied","Data":"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"}
Sep 08 00:21:43.245142 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.244804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747" event={"ID":"bc491686-a4c9-4eec-8daf-1c1f4be9a200","Type":"ContainerDied","Data":"0e9c300262dd41ce92b872714cc509399060334304a17e8f3355eb233d8e56aa"}
Sep 08 00:21:43.245142 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.244833    2580 scope.go:117] "RemoveContainer" containerID="60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"
Sep 08 00:21:43.246295 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.246267    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" containerID="503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c" exitCode=0
Sep 08 00:21:43.246393 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.246308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"
Sep 08 00:21:43.246393 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.246353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" event={"ID":"c9aa4328-49d6-4b58-bbab-ea3fb6278510","Type":"ContainerDied","Data":"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"}
Sep 08 00:21:43.246508 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.246402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275" event={"ID":"c9aa4328-49d6-4b58-bbab-ea3fb6278510","Type":"ContainerDied","Data":"1eeb92cecffe64972f079174e04230dca7e89afeed746a2cf5c93d4fac627974"}
Sep 08 00:21:43.247845 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.247819    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad263fea-b60a-4c20-b876-30c4b8d5560e" containerID="492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04" exitCode=0
Sep 08 00:21:43.247936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.247859    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"
Sep 08 00:21:43.247936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.247875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" event={"ID":"ad263fea-b60a-4c20-b876-30c4b8d5560e","Type":"ContainerDied","Data":"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"}
Sep 08 00:21:43.248049 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.247934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd" event={"ID":"ad263fea-b60a-4c20-b876-30c4b8d5560e","Type":"ContainerDied","Data":"a278767df1dd3ac599d4a490066de4ed97affa998b540b86f4655cdabe91ea56"}
Sep 08 00:21:43.255648 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.255626    2580 scope.go:117] "RemoveContainer" containerID="60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"
Sep 08 00:21:43.255943 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:43.255913    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f\": container with ID starting with 60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f not found: ID does not exist" containerID="60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"
Sep 08 00:21:43.256042 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.255950    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f"} err="failed to get container status \"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f\": rpc error: code = NotFound desc = could not find container \"60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f\": container with ID starting with 60a86165d9661981c626cd47cc21e8f88d6235c098848362f70804f6e056868f not found: ID does not exist"
Sep 08 00:21:43.256042 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.255973    2580 scope.go:117] "RemoveContainer" containerID="503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"
Sep 08 00:21:43.264353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.264336    2580 scope.go:117] "RemoveContainer" containerID="503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"
Sep 08 00:21:43.264640 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:43.264621    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c\": container with ID starting with 503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c not found: ID does not exist" containerID="503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"
Sep 08 00:21:43.264709 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.264645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c"} err="failed to get container status \"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c\": rpc error: code = NotFound desc = could not find container \"503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c\": container with ID starting with 503d722548795544d8d822fb44fe77d656fef968efed041666257efc6702056c not found: ID does not exist"
Sep 08 00:21:43.264709 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.264660    2580 scope.go:117] "RemoveContainer" containerID="492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"
Sep 08 00:21:43.275317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.273164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:21:43.275317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.273458    2580 scope.go:117] "RemoveContainer" containerID="492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"
Sep 08 00:21:43.275317 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:43.274154    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04\": container with ID starting with 492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04 not found: ID does not exist" containerID="492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"
Sep 08 00:21:43.275317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.274182    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04"} err="failed to get container status \"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04\": rpc error: code = NotFound desc = could not find container \"492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04\": container with ID starting with 492dc8f72a5e60e495fb68ca56a261d3a78c4b65557a535dc2e30742eee29a04 not found: ID does not exist"
Sep 08 00:21:43.277997 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.277976    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-75e439-5fc8758f5-cm275"]
Sep 08 00:21:43.286184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.286163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:21:43.289398 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.289376    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-75e439-84876c98b5-s4747"]
Sep 08 00:21:43.298502 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.298477    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:21:43.301860 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:43.301838    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-75e439-6986fd8954-j9hwd"]
Sep 08 00:21:44.330293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:44.330250    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad263fea-b60a-4c20-b876-30c4b8d5560e" path="/var/lib/kubelet/pods/ad263fea-b60a-4c20-b876-30c4b8d5560e/volumes"
Sep 08 00:21:44.330800 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:44.330786    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" path="/var/lib/kubelet/pods/bc491686-a4c9-4eec-8daf-1c1f4be9a200/volumes"
Sep 08 00:21:44.331203 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:44.331174    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" path="/var/lib/kubelet/pods/c9aa4328-49d6-4b58-bbab-ea3fb6278510/volumes"
Sep 08 00:21:48.253549 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:48.253512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:21:52.076007 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.075968    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:21:52.076507 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.076167    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" podUID="0380e69d-0527-4b29-9414-0f5a5322429c" containerName="maas-api" containerID="cri-o://8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950" gracePeriod=30
Sep 08 00:21:52.239519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.239483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:21:52.540907 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.540880    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:21:52.715627 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.715596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca\") pod \"0380e69d-0527-4b29-9414-0f5a5322429c\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") "
Sep 08 00:21:52.715829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.715642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs\") pod \"0380e69d-0527-4b29-9414-0f5a5322429c\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") "
Sep 08 00:21:52.715829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.715669    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmtrk\" (UniqueName: \"kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk\") pod \"0380e69d-0527-4b29-9414-0f5a5322429c\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") "
Sep 08 00:21:52.715829 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.715721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls\") pod \"0380e69d-0527-4b29-9414-0f5a5322429c\" (UID: \"0380e69d-0527-4b29-9414-0f5a5322429c\") "
Sep 08 00:21:52.715998 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.715972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0380e69d-0527-4b29-9414-0f5a5322429c" (UID: "0380e69d-0527-4b29-9414-0f5a5322429c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:21:52.717807 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.717777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0380e69d-0527-4b29-9414-0f5a5322429c" (UID: "0380e69d-0527-4b29-9414-0f5a5322429c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:21:52.717965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.717840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0380e69d-0527-4b29-9414-0f5a5322429c" (UID: "0380e69d-0527-4b29-9414-0f5a5322429c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:21:52.717965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.717908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk" (OuterVolumeSpecName: "kube-api-access-nmtrk") pod "0380e69d-0527-4b29-9414-0f5a5322429c" (UID: "0380e69d-0527-4b29-9414-0f5a5322429c"). InnerVolumeSpecName "kube-api-access-nmtrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:21:52.817301 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.817260    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:52.817301 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.817292    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0380e69d-0527-4b29-9414-0f5a5322429c-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:52.817301 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.817302    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0380e69d-0527-4b29-9414-0f5a5322429c-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:52.817301 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:52.817311    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmtrk\" (UniqueName: \"kubernetes.io/projected/0380e69d-0527-4b29-9414-0f5a5322429c-kube-api-access-nmtrk\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:21:53.304824 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.304784    2580 generic.go:356] "Generic (PLEG): container finished" podID="0380e69d-0527-4b29-9414-0f5a5322429c" containerID="8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950" exitCode=0
Sep 08 00:21:53.305194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.304829    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"
Sep 08 00:21:53.305194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.304868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" event={"ID":"0380e69d-0527-4b29-9414-0f5a5322429c","Type":"ContainerDied","Data":"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"}
Sep 08 00:21:53.305194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.304923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw" event={"ID":"0380e69d-0527-4b29-9414-0f5a5322429c","Type":"ContainerDied","Data":"dece0f616e4d46a1dc2beaa19d5c35eb126d61fb550533d473ca6708d4b26a90"}
Sep 08 00:21:53.305194 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.304946    2580 scope.go:117] "RemoveContainer" containerID="8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"
Sep 08 00:21:53.321123 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.321099    2580 scope.go:117] "RemoveContainer" containerID="8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"
Sep 08 00:21:53.321474 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:21:53.321453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950\": container with ID starting with 8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950 not found: ID does not exist" containerID="8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"
Sep 08 00:21:53.321519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.321482    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950"} err="failed to get container status \"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950\": rpc error: code = NotFound desc = could not find container \"8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950\": container with ID starting with 8012f0fd43d591202490eece0451cec017efb4a099d461dfc2693328b2af0950 not found: ID does not exist"
Sep 08 00:21:53.332276 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.332247    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:21:53.335683 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:53.335657    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4374d7-64587d6cc8-nwlbw"]
Sep 08 00:21:54.330628 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:21:54.330593    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0380e69d-0527-4b29-9414-0f5a5322429c" path="/var/lib/kubelet/pods/0380e69d-0527-4b29-9414-0f5a5322429c/volumes"
Sep 08 00:22:09.731565 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:09.731517    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:22:09.732068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:09.731787    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" podUID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" containerName="maas-api" containerID="cri-o://4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0" gracePeriod=30
Sep 08 00:22:09.973936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:09.973909    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:22:10.070787 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.070693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs\") pod \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") "
Sep 08 00:22:10.070954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.070793    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca\") pod \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") "
Sep 08 00:22:10.070954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.070823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") pod \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") "
Sep 08 00:22:10.070954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.070930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc9z7\" (UniqueName: \"kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7\") pod \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\" (UID: \"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc\") "
Sep 08 00:22:10.071129 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.071093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" (UID: "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:22:10.071236 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.071220    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:10.072934 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.072900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" (UID: "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:22:10.073032 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.072946    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" (UID: "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:22:10.073032 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.072975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7" (OuterVolumeSpecName: "kube-api-access-rc9z7") pod "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" (UID: "c86c20c6-5c48-4812-8c4e-57eeeb8d21bc"). InnerVolumeSpecName "kube-api-access-rc9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:22:10.172122 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.172084    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:10.172122 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.172117    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc9z7\" (UniqueName: \"kubernetes.io/projected/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-kube-api-access-rc9z7\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:10.172311 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.172131    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:10.381054 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.381018    2580 generic.go:356] "Generic (PLEG): container finished" podID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" containerID="4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0" exitCode=0
Sep 08 00:22:10.381203 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.381065    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"
Sep 08 00:22:10.381203 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.381068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" event={"ID":"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc","Type":"ContainerDied","Data":"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"}
Sep 08 00:22:10.381316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.381200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm" event={"ID":"c86c20c6-5c48-4812-8c4e-57eeeb8d21bc","Type":"ContainerDied","Data":"3b6c2713d5be1f23d5bdf353ae1986e7172fd30f515cae14e37bf1bce141b673"}
Sep 08 00:22:10.381316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.381233    2580 scope.go:117] "RemoveContainer" containerID="4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"
Sep 08 00:22:10.389685 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.389660    2580 scope.go:117] "RemoveContainer" containerID="4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"
Sep 08 00:22:10.389954 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:10.389934    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0\": container with ID starting with 4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0 not found: ID does not exist" containerID="4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"
Sep 08 00:22:10.390013 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.389962    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0"} err="failed to get container status \"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0\": rpc error: code = NotFound desc = could not find container \"4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0\": container with ID starting with 4b4d73a3b39686f72c94110e687ecdf40550e8e3b72bf39029d5d3fa624310a0 not found: ID does not exist"
Sep 08 00:22:10.396079 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.396045    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:22:10.399656 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:10.399634    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c4541b-fc548c6b9-p4dmm"]
Sep 08 00:22:12.330710 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:12.330676    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" path="/var/lib/kubelet/pods/c86c20c6-5c48-4812-8c4e-57eeeb8d21bc/volumes"
Sep 08 00:22:21.023828 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.023795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:22:21.024213 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024199    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0380e69d-0527-4b29-9414-0f5a5322429c" containerName="maas-api"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024215    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0380e69d-0527-4b29-9414-0f5a5322429c" containerName="maas-api"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024223    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" containerName="maas-api"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024232    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" containerName="maas-api"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024241    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad263fea-b60a-4c20-b876-30c4b8d5560e" containerName="payload-pre-processing"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024248    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad263fea-b60a-4c20-b876-30c4b8d5560e" containerName="payload-pre-processing"
Sep 08 00:22:21.024260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024259    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" containerName="maas-api"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024264    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" containerName="maas-api"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024277    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" containerName="payload-processing"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024282    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" containerName="payload-processing"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024337    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc491686-a4c9-4eec-8daf-1c1f4be9a200" containerName="payload-processing"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024345    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86c20c6-5c48-4812-8c4e-57eeeb8d21bc" containerName="maas-api"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024353    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9aa4328-49d6-4b58-bbab-ea3fb6278510" containerName="maas-api"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024359    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0380e69d-0527-4b29-9414-0f5a5322429c" containerName="maas-api"
Sep 08 00:22:21.024481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.024365    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad263fea-b60a-4c20-b876-30c4b8d5560e" containerName="payload-pre-processing"
Sep 08 00:22:21.027666 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.027643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.029780 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.029750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-8978e7-serving-cert\""
Sep 08 00:22:21.040405 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.040377    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:22:21.067316 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.067273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.067497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.067338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.067497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.067414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.067497 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.067454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9t7s\" (UniqueName: \"kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.169012 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.168975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.169172 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.169024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.169172 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.169059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.169172 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.169081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9t7s\" (UniqueName: \"kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.169276 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:21.169186    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-serving-cert: secret "maas-api-e2e-shared-a-w3-8978e7-serving-cert" not found
Sep 08 00:22:21.169276 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:21.169273    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls podName:74ade461-58f7-479c-812d-36a0d3381da3 nodeName:}" failed. No retries permitted until 2026-09-08 00:22:21.669255731 +0000 UTC m=+1173.841520598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls") pod "maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" (UID: "74ade461-58f7-479c-812d-36a0d3381da3") : secret "maas-api-e2e-shared-a-w3-8978e7-serving-cert" not found
Sep 08 00:22:21.169657 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.169637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.171373 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.171355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.176880 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.176855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9t7s\" (UniqueName: \"kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.673830 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.673778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.676454 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.676422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:21.799743 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.799704    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:22:21.799979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.799951    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" podUID="ef9eda10-785c-4256-99ab-21ccbb615dc8" containerName="maas-api" containerID="cri-o://f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a" gracePeriod=30
Sep 08 00:22:21.847975 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.847937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:22:21.848217 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.848189    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" podUID="7636c0ca-670d-4266-a6fc-b996f7529d85" containerName="payload-processing" containerID="cri-o://24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5" gracePeriod=30
Sep 08 00:22:21.940677 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:21.940593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:22.133748 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.133719    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:22:22.143343 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.143321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:22:22.178968 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.178941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") pod \"ef9eda10-785c-4256-99ab-21ccbb615dc8\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") "
Sep 08 00:22:22.179133 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.178984    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs\") pod \"ef9eda10-785c-4256-99ab-21ccbb615dc8\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") "
Sep 08 00:22:22.179133 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.179019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kllvz\" (UniqueName: \"kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz\") pod \"ef9eda10-785c-4256-99ab-21ccbb615dc8\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") "
Sep 08 00:22:22.179265 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.179148    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca\") pod \"ef9eda10-785c-4256-99ab-21ccbb615dc8\" (UID: \"ef9eda10-785c-4256-99ab-21ccbb615dc8\") "
Sep 08 00:22:22.179693 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.179642    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef9eda10-785c-4256-99ab-21ccbb615dc8" (UID: "ef9eda10-785c-4256-99ab-21ccbb615dc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:22:22.181624 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.181586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef9eda10-785c-4256-99ab-21ccbb615dc8" (UID: "ef9eda10-785c-4256-99ab-21ccbb615dc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:22:22.181734 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.181600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz" (OuterVolumeSpecName: "kube-api-access-kllvz") pod "ef9eda10-785c-4256-99ab-21ccbb615dc8" (UID: "ef9eda10-785c-4256-99ab-21ccbb615dc8"). InnerVolumeSpecName "kube-api-access-kllvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:22:22.181734 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.181692    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef9eda10-785c-4256-99ab-21ccbb615dc8" (UID: "ef9eda10-785c-4256-99ab-21ccbb615dc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:22:22.214913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.214880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:22:22.217926 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:22:22.217900    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ade461_58f7_479c_812d_36a0d3381da3.slice/crio-0227b32778ab10195cce1607f5cba5d5b7cbcc7179974be5c02d206e3526b0e5 WatchSource:0}: Error finding container 0227b32778ab10195cce1607f5cba5d5b7cbcc7179974be5c02d206e3526b0e5: Status 404 returned error can't find the container with id 0227b32778ab10195cce1607f5cba5d5b7cbcc7179974be5c02d206e3526b0e5
Sep 08 00:22:22.280054 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume\") pod \"7636c0ca-670d-4266-a6fc-b996f7529d85\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") "
Sep 08 00:22:22.280220 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nq57\" (UniqueName: \"kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57\") pod \"7636c0ca-670d-4266-a6fc-b996f7529d85\" (UID: \"7636c0ca-670d-4266-a6fc-b996f7529d85\") "
Sep 08 00:22:22.280878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280403    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.280878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280429    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kllvz\" (UniqueName: \"kubernetes.io/projected/ef9eda10-785c-4256-99ab-21ccbb615dc8-kube-api-access-kllvz\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.280878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280454    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9eda10-785c-4256-99ab-21ccbb615dc8-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.280878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280480    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9eda10-785c-4256-99ab-21ccbb615dc8-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.280878 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.280473    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7636c0ca-670d-4266-a6fc-b996f7529d85" (UID: "7636c0ca-670d-4266-a6fc-b996f7529d85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:22:22.282293 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.282270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57" (OuterVolumeSpecName: "kube-api-access-6nq57") pod "7636c0ca-670d-4266-a6fc-b996f7529d85" (UID: "7636c0ca-670d-4266-a6fc-b996f7529d85"). InnerVolumeSpecName "kube-api-access-6nq57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:22:22.380989 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.380956    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7636c0ca-670d-4266-a6fc-b996f7529d85-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.381149 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.380998    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nq57\" (UniqueName: \"kubernetes.io/projected/7636c0ca-670d-4266-a6fc-b996f7529d85-kube-api-access-6nq57\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:22:22.437014 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.436976    2580 generic.go:356] "Generic (PLEG): container finished" podID="ef9eda10-785c-4256-99ab-21ccbb615dc8" containerID="f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a" exitCode=0
Sep 08 00:22:22.437182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.437020    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"
Sep 08 00:22:22.437182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.437055    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" event={"ID":"ef9eda10-785c-4256-99ab-21ccbb615dc8","Type":"ContainerDied","Data":"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"}
Sep 08 00:22:22.437182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.437106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz" event={"ID":"ef9eda10-785c-4256-99ab-21ccbb615dc8","Type":"ContainerDied","Data":"314013b95ef627d67f32c561a9833b93f954de59f21ed80eb0528623af64c7ff"}
Sep 08 00:22:22.437182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.437136    2580 scope.go:117] "RemoveContainer" containerID="f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"
Sep 08 00:22:22.438521 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.438492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" event={"ID":"74ade461-58f7-479c-812d-36a0d3381da3","Type":"ContainerStarted","Data":"0227b32778ab10195cce1607f5cba5d5b7cbcc7179974be5c02d206e3526b0e5"}
Sep 08 00:22:22.440206 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.440180    2580 generic.go:356] "Generic (PLEG): container finished" podID="7636c0ca-670d-4266-a6fc-b996f7529d85" containerID="24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5" exitCode=0
Sep 08 00:22:22.440294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.440216    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"
Sep 08 00:22:22.440294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.440237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" event={"ID":"7636c0ca-670d-4266-a6fc-b996f7529d85","Type":"ContainerDied","Data":"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"}
Sep 08 00:22:22.440294 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.440268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m" event={"ID":"7636c0ca-670d-4266-a6fc-b996f7529d85","Type":"ContainerDied","Data":"e87d261675f0dfa7cbd0e0a3fc54ec04a81a5a20924b4a9696589353343ae7e9"}
Sep 08 00:22:22.446395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.446378    2580 scope.go:117] "RemoveContainer" containerID="f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"
Sep 08 00:22:22.446676 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:22.446654    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a\": container with ID starting with f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a not found: ID does not exist" containerID="f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"
Sep 08 00:22:22.446729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.446684    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a"} err="failed to get container status \"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a\": rpc error: code = NotFound desc = could not find container \"f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a\": container with ID starting with f26ef5112a64afa70100b5940dfde69b843e1937338071b9691acd53bf95e76a not found: ID does not exist"
Sep 08 00:22:22.446729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.446701    2580 scope.go:117] "RemoveContainer" containerID="24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"
Sep 08 00:22:22.450920 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.450896    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:22:22.453301 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.453279    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e250ebf1-78d5bc6fd5-n88rz"]
Sep 08 00:22:22.455734 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.455717    2580 scope.go:117] "RemoveContainer" containerID="24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"
Sep 08 00:22:22.456023 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:22.456005    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5\": container with ID starting with 24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5 not found: ID does not exist" containerID="24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"
Sep 08 00:22:22.456075 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.456030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5"} err="failed to get container status \"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5\": rpc error: code = NotFound desc = could not find container \"24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5\": container with ID starting with 24269c52a1e57e77ac43f91377619a46c5cadc5f36e6b6835298a9c8ce66dad5 not found: ID does not exist"
Sep 08 00:22:22.462381 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.462359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:22:22.466225 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:22.466205    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e250ebf1-79776648b9-fxc5m"]
Sep 08 00:22:23.452115 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:23.452079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" event={"ID":"74ade461-58f7-479c-812d-36a0d3381da3","Type":"ContainerStarted","Data":"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"}
Sep 08 00:22:23.452597 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:23.452194    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:23.470444 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:23.470390    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" podStartSLOduration=2.081744103 podStartE2EDuration="2.470351852s" totalImagesPullingTime="388.607749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:22:23.468575751 +0000 UTC m=+1175.640840634" watchObservedRunningTime="2026-09-08 00:22:23.470351852 +0000 UTC m=+1175.642616735"
Sep 08 00:22:24.329465 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:24.329428    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7636c0ca-670d-4266-a6fc-b996f7529d85" path="/var/lib/kubelet/pods/7636c0ca-670d-4266-a6fc-b996f7529d85/volumes"
Sep 08 00:22:24.329848 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:24.329835    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9eda10-785c-4256-99ab-21ccbb615dc8" path="/var/lib/kubelet/pods/ef9eda10-785c-4256-99ab-21ccbb615dc8/volumes"
Sep 08 00:22:29.463867 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:29.463833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:22:46.246043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246002    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246441    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9eda10-785c-4256-99ab-21ccbb615dc8" containerName="maas-api"
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9eda10-785c-4256-99ab-21ccbb615dc8" containerName="maas-api"
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246483    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7636c0ca-670d-4266-a6fc-b996f7529d85" containerName="payload-processing"
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246491    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7636c0ca-670d-4266-a6fc-b996f7529d85" containerName="payload-processing"
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246609    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7636c0ca-670d-4266-a6fc-b996f7529d85" containerName="payload-processing"
Sep 08 00:22:46.246706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.246621    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9eda10-785c-4256-99ab-21ccbb615dc8" containerName="maas-api"
Sep 08 00:22:46.249751 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.249725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.251966 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.251939    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c6a703-serving-cert\""
Sep 08 00:22:46.258990 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.258951    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:22:46.405765 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.405724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.405765 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.405769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2qp\" (UniqueName: \"kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.405973 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.405810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.405973 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.405912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.506665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.506561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2qp\" (UniqueName: \"kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.506665 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.506615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.506924 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.506715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.506924 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.506777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.506924 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:46.506780    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-serving-cert: secret "maas-api-e2e-shared-a-w2-c6a703-serving-cert" not found
Sep 08 00:22:46.506924 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:22:46.506854    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls podName:24323acc-0617-4f94-ba0c-148c9ac62b99 nodeName:}" failed. No retries permitted until 2026-09-08 00:22:47.006837524 +0000 UTC m=+1199.179102404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" (UID: "24323acc-0617-4f94-ba0c-148c9ac62b99") : secret "maas-api-e2e-shared-a-w2-c6a703-serving-cert" not found
Sep 08 00:22:46.507482 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.507457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.509118 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.509100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.516880 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.516853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2qp\" (UniqueName: \"kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:46.876931 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.876834    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:22:46.880574 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.880521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:46.882622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.882592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c6a703\""
Sep 08 00:22:46.890999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:46.890964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:22:47.011840 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.011800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vrk\" (UniqueName: \"kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.012051 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.011924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.012051 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.011974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:47.014270 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.014253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:47.113360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.113318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vrk\" (UniqueName: \"kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.113582 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.113398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.114160 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.114131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.120832 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.120804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vrk\" (UniqueName: \"kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk\") pod \"payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.162978 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.162944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:47.193248 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.193210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.436636 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.436587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:22:47.439183 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:22:47.439157    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24323acc_0617_4f94_ba0c_148c9ac62b99.slice/crio-b7b81e3befc9f8f2095d0971d319a455af955341f15b63d61931f15c85c07357 WatchSource:0}: Error finding container b7b81e3befc9f8f2095d0971d319a455af955341f15b63d61931f15c85c07357: Status 404 returned error can't find the container with id b7b81e3befc9f8f2095d0971d319a455af955341f15b63d61931f15c85c07357
Sep 08 00:22:47.474958 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.474931    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:22:47.477403 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:22:47.477365    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e467e0_e7ed_40fb_b801_d5a386c93cfe.slice/crio-d01cba60719ae9198c864c5a2c9b28b9e36c471625f525875e8261b93b5e4bf8 WatchSource:0}: Error finding container d01cba60719ae9198c864c5a2c9b28b9e36c471625f525875e8261b93b5e4bf8: Status 404 returned error can't find the container with id d01cba60719ae9198c864c5a2c9b28b9e36c471625f525875e8261b93b5e4bf8
Sep 08 00:22:47.565871 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.565825    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" event={"ID":"51e467e0-e7ed-40fb-b801-d5a386c93cfe","Type":"ContainerStarted","Data":"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"}
Sep 08 00:22:47.566047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.565891    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" event={"ID":"51e467e0-e7ed-40fb-b801-d5a386c93cfe","Type":"ContainerStarted","Data":"d01cba60719ae9198c864c5a2c9b28b9e36c471625f525875e8261b93b5e4bf8"}
Sep 08 00:22:47.566047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.565922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:22:47.567404 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.567376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" event={"ID":"24323acc-0617-4f94-ba0c-148c9ac62b99","Type":"ContainerStarted","Data":"b7b81e3befc9f8f2095d0971d319a455af955341f15b63d61931f15c85c07357"}
Sep 08 00:22:47.581644 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:47.581590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" podStartSLOduration=1.581572447 podStartE2EDuration="1.581572447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:22:47.579599348 +0000 UTC m=+1199.751864233" watchObservedRunningTime="2026-09-08 00:22:47.581572447 +0000 UTC m=+1199.753837330"
Sep 08 00:22:48.573623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:48.573571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" event={"ID":"24323acc-0617-4f94-ba0c-148c9ac62b99","Type":"ContainerStarted","Data":"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"}
Sep 08 00:22:48.603572 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:48.603503    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" podStartSLOduration=2.108462196 podStartE2EDuration="2.603462069s" totalImagesPullingTime="494.999873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:22:48.602652202 +0000 UTC m=+1200.774917088" watchObservedRunningTime="2026-09-08 00:22:48.603462069 +0000 UTC m=+1200.775726953"
Sep 08 00:22:49.265362 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.265318    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:22:49.270964 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.270934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.273506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.273389    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-b8c8cd-serving-cert\""
Sep 08 00:22:49.280587 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.280335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:22:49.337411 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.337374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.337411 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.337414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.337687 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.337442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7p4h\" (UniqueName: \"kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.337687 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.337595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.438247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.438211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.438442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.438289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.438442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.438320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.438442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.438354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7p4h\" (UniqueName: \"kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.439048 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.439016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.440941 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.440913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.441038 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.440935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.445581 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.445554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7p4h\" (UniqueName: \"kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h\") pod \"maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.578483 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.578393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:49.585710 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.585676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:49.860148 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.860117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:22:49.862866 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:22:49.862827    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a02dd05_affd_4ba1_9412_a5ab0108cb9f.slice/crio-22c83f6e787329f1e46c56c03331fc4c62b690251aa489a1ab396396ba4cb90a WatchSource:0}: Error finding container 22c83f6e787329f1e46c56c03331fc4c62b690251aa489a1ab396396ba4cb90a: Status 404 returned error can't find the container with id 22c83f6e787329f1e46c56c03331fc4c62b690251aa489a1ab396396ba4cb90a
Sep 08 00:22:49.910564 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.910514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:22:49.915043 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.915022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:49.917071 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.917048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b8c8cd-dockercfg-2jxc9\""
Sep 08 00:22:49.917202 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.917067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b8c8cd\""
Sep 08 00:22:49.922146 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.922121    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:22:49.941898 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.941864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4f5t\" (UniqueName: \"kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:49.942062 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:49.941924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.042615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.042579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4f5t\" (UniqueName: \"kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.042813 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.042644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.043330 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.043307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.052135 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.052111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4f5t\" (UniqueName: \"kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t\") pod \"payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.227362 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.227327    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.503908 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.503789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:22:50.506696 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:22:50.506668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef2c13b_a5e5_4068_a1b1_1bf2fe9da1a1.slice/crio-bce9c5687e1450845720fec39f269eea831462c9ca84286a5cead1b30f07d519 WatchSource:0}: Error finding container bce9c5687e1450845720fec39f269eea831462c9ca84286a5cead1b30f07d519: Status 404 returned error can't find the container with id bce9c5687e1450845720fec39f269eea831462c9ca84286a5cead1b30f07d519
Sep 08 00:22:50.587928 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.587886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" event={"ID":"3a02dd05-affd-4ba1-9412-a5ab0108cb9f","Type":"ContainerStarted","Data":"bc3c7d841454f0ef5de04ec8c17fb65e7dc32f64afae7da6669e09bc7d741650"}
Sep 08 00:22:50.588368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.588066    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:50.588368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.588109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" event={"ID":"3a02dd05-affd-4ba1-9412-a5ab0108cb9f","Type":"ContainerStarted","Data":"22c83f6e787329f1e46c56c03331fc4c62b690251aa489a1ab396396ba4cb90a"}
Sep 08 00:22:50.589703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.589673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" event={"ID":"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1","Type":"ContainerStarted","Data":"ec39b9dc44ac2c8cbe52c7037eed59e1a6f48ec50d71cc10ec877e8b42640239"}
Sep 08 00:22:50.589859 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.589730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" event={"ID":"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1","Type":"ContainerStarted","Data":"bce9c5687e1450845720fec39f269eea831462c9ca84286a5cead1b30f07d519"}
Sep 08 00:22:50.589933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.589871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:22:50.604188 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.604127    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" podStartSLOduration=1.269886799 podStartE2EDuration="1.604106934s" totalImagesPullingTime="334.220135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:22:50.603390811 +0000 UTC m=+1202.775655708" watchObservedRunningTime="2026-09-08 00:22:50.604106934 +0000 UTC m=+1202.776371819"
Sep 08 00:22:50.622086 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:50.621729    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" podStartSLOduration=1.6216691060000001 podStartE2EDuration="1.621669106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:22:50.621060466 +0000 UTC m=+1202.793325352" watchObservedRunningTime="2026-09-08 00:22:50.621669106 +0000 UTC m=+1202.793933991"
Sep 08 00:22:55.595211 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:55.595171    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:22:56.600434 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:56.600403    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:22:58.575987 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:22:58.575951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:23:01.597125 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:01.597087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:23:14.344802 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.344764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:23:14.349781 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.349758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.351861 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.351833    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e1e91c-serving-cert\""
Sep 08 00:23:14.359881 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.359854    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:23:14.467965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.467929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.468140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.467997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.468140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.468037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrkl\" (UniqueName: \"kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.468140 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.468063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.569322 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.569283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.569489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.569353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.569489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.569394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrkl\" (UniqueName: \"kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.569489 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.569418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.569988 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.569962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.572032 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.572007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.572126 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.572007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.578520 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.578498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrkl\" (UniqueName: \"kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl\") pod \"maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.663469 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.663433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:14.938511 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:14.938485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:23:14.941265 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:23:14.941235    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2753b_08b3_4b94_aa47_44da37b88ae3.slice/crio-cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0 WatchSource:0}: Error finding container cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0: Status 404 returned error can't find the container with id cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0
Sep 08 00:23:15.707086 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:15.707045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" event={"ID":"bca2753b-08b3-4b94-aa47-44da37b88ae3","Type":"ContainerStarted","Data":"a7fe71cc6b6a96ac32d683c2dd391548fd8583d43596343969378d555e5335de"}
Sep 08 00:23:15.707451 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:15.707101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" event={"ID":"bca2753b-08b3-4b94-aa47-44da37b88ae3","Type":"ContainerStarted","Data":"cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0"}
Sep 08 00:23:15.707451 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:15.707157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:15.723850 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:15.723793    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" podStartSLOduration=1.420145339 podStartE2EDuration="1.723775941s" totalImagesPullingTime="303.630602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:23:15.721354588 +0000 UTC m=+1227.893619471" watchObservedRunningTime="2026-09-08 00:23:15.723775941 +0000 UTC m=+1227.896040825"
Sep 08 00:23:21.716638 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:21.716606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:23:55.724026 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.723986    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:23:55.727937 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.727917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.730202 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.730179    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b1c04f8f-serving-cert\""
Sep 08 00:23:55.735868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.735480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:23:55.858827 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.858790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.859003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.858854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.859003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.858945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.859003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.858978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69v5\" (UniqueName: \"kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.960252 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.960212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.960447 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.960275    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.960447 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.960297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69v5\" (UniqueName: \"kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.960447 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.960349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.960447 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:23:55.960443    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-serving-cert: secret "maas-api-e2e-mt-b1c04f8f-serving-cert" not found
Sep 08 00:23:55.960626 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:23:55.960513    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls podName:25366a7c-b6cf-45e0-b378-516c29f1b75e nodeName:}" failed. No retries permitted until 2026-09-08 00:23:56.460493788 +0000 UTC m=+1268.632758652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls") pod "maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" (UID: "25366a7c-b6cf-45e0-b378-516c29f1b75e") : secret "maas-api-e2e-mt-b1c04f8f-serving-cert" not found
Sep 08 00:23:55.960972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.960948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.962794 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.962773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:55.969720 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:55.969694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69v5\" (UniqueName: \"kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:56.466022 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:56.465982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:56.468368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:56.468340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") pod \"maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:56.640240 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:56.640200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:56.917111 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:56.917086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:23:56.919579 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:23:56.919548    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25366a7c_b6cf_45e0_b378_516c29f1b75e.slice/crio-fd1cf2899494a5125cb86b4a15fffdda238fc77ce905e3b2865fc76b8bf0ea7c WatchSource:0}: Error finding container fd1cf2899494a5125cb86b4a15fffdda238fc77ce905e3b2865fc76b8bf0ea7c: Status 404 returned error can't find the container with id fd1cf2899494a5125cb86b4a15fffdda238fc77ce905e3b2865fc76b8bf0ea7c
Sep 08 00:23:57.901490 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:57.901454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" event={"ID":"25366a7c-b6cf-45e0-b378-516c29f1b75e","Type":"ContainerStarted","Data":"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"}
Sep 08 00:23:57.901689 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:57.901558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" event={"ID":"25366a7c-b6cf-45e0-b378-516c29f1b75e","Type":"ContainerStarted","Data":"fd1cf2899494a5125cb86b4a15fffdda238fc77ce905e3b2865fc76b8bf0ea7c"}
Sep 08 00:23:57.901689 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:57.901588    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:23:57.920725 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:23:57.920676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" podStartSLOduration=2.406766122 podStartE2EDuration="2.920658948s" totalImagesPullingTime="513.892826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:23:57.918281279 +0000 UTC m=+1270.090546165" watchObservedRunningTime="2026-09-08 00:23:57.920658948 +0000 UTC m=+1270.092923831"
Sep 08 00:24:03.911029 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:24:03.911000    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:25:24.440244 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.440163    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:25:24.440779 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.440408    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" podUID="25366a7c-b6cf-45e0-b378-516c29f1b75e" containerName="maas-api" containerID="cri-o://b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7" gracePeriod=30
Sep 08 00:25:24.686869 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.686840    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:25:24.803261 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.803171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") pod \"25366a7c-b6cf-45e0-b378-516c29f1b75e\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") "
Sep 08 00:25:24.803261 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.803221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca\") pod \"25366a7c-b6cf-45e0-b378-516c29f1b75e\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") "
Sep 08 00:25:24.803261 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.803251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x69v5\" (UniqueName: \"kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5\") pod \"25366a7c-b6cf-45e0-b378-516c29f1b75e\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") "
Sep 08 00:25:24.803583 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.803309    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs\") pod \"25366a7c-b6cf-45e0-b378-516c29f1b75e\" (UID: \"25366a7c-b6cf-45e0-b378-516c29f1b75e\") "
Sep 08 00:25:24.803703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.803675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25366a7c-b6cf-45e0-b378-516c29f1b75e" (UID: "25366a7c-b6cf-45e0-b378-516c29f1b75e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:25:24.805272 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.805244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25366a7c-b6cf-45e0-b378-516c29f1b75e" (UID: "25366a7c-b6cf-45e0-b378-516c29f1b75e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:25:24.805377 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.805322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5" (OuterVolumeSpecName: "kube-api-access-x69v5") pod "25366a7c-b6cf-45e0-b378-516c29f1b75e" (UID: "25366a7c-b6cf-45e0-b378-516c29f1b75e"). InnerVolumeSpecName "kube-api-access-x69v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:25:24.805423 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.805405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "25366a7c-b6cf-45e0-b378-516c29f1b75e" (UID: "25366a7c-b6cf-45e0-b378-516c29f1b75e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:25:24.905152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.905115    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:25:24.905152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.905145    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25366a7c-b6cf-45e0-b378-516c29f1b75e-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:25:24.905152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.905156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x69v5\" (UniqueName: \"kubernetes.io/projected/25366a7c-b6cf-45e0-b378-516c29f1b75e-kube-api-access-x69v5\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:25:24.905397 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:24.905165    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25366a7c-b6cf-45e0-b378-516c29f1b75e-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:25:25.307768 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.307730    2580 generic.go:356] "Generic (PLEG): container finished" podID="25366a7c-b6cf-45e0-b378-516c29f1b75e" containerID="b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7" exitCode=0
Sep 08 00:25:25.307972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.307772    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" event={"ID":"25366a7c-b6cf-45e0-b378-516c29f1b75e","Type":"ContainerDied","Data":"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"}
Sep 08 00:25:25.307972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.307819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9" event={"ID":"25366a7c-b6cf-45e0-b378-516c29f1b75e","Type":"ContainerDied","Data":"fd1cf2899494a5125cb86b4a15fffdda238fc77ce905e3b2865fc76b8bf0ea7c"}
Sep 08 00:25:25.307972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.307848    2580 scope.go:117] "RemoveContainer" containerID="b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"
Sep 08 00:25:25.307972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.307775    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"
Sep 08 00:25:25.317282 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.317262    2580 scope.go:117] "RemoveContainer" containerID="b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"
Sep 08 00:25:25.317584 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:25:25.317561    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7\": container with ID starting with b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7 not found: ID does not exist" containerID="b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"
Sep 08 00:25:25.317676 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.317590    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7"} err="failed to get container status \"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7\": rpc error: code = NotFound desc = could not find container \"b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7\": container with ID starting with b0529818299b08c87e2eae316081f5ee5ef974a35a959e7e6e4846640ec16da7 not found: ID does not exist"
Sep 08 00:25:25.327256 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.327233    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:25:25.333297 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:25.333275    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1c04f8f-84d9488b59-xqmk9"]
Sep 08 00:25:26.329517 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:25:26.329481    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25366a7c-b6cf-45e0-b378-516c29f1b75e" path="/var/lib/kubelet/pods/25366a7c-b6cf-45e0-b378-516c29f1b75e/volumes"
Sep 08 00:27:27.155154 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.155120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:27:27.155739 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.155718    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25366a7c-b6cf-45e0-b378-516c29f1b75e" containerName="maas-api"
Sep 08 00:27:27.155820 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.155741    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25366a7c-b6cf-45e0-b378-516c29f1b75e" containerName="maas-api"
Sep 08 00:27:27.155874 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.155840    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25366a7c-b6cf-45e0-b378-516c29f1b75e" containerName="maas-api"
Sep 08 00:27:27.159005 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.158983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.161577 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.161556    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-93c8ee-serving-cert\""
Sep 08 00:27:27.177012 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.176988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:27:27.344979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.344944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.344979 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.344980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.345179 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.345010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqbt\" (UniqueName: \"kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.345179 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.345092    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.445655 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.445564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.445655 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.445608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.445655 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.445643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqbt\" (UniqueName: \"kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.445941 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.445696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.446365 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.446339    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.448053 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.448023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.448152 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.448088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.453745 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.453721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqbt\" (UniqueName: \"kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt\") pod \"maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.468805 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.468782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:27.741963 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.741873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:27:27.744946 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:27:27.744914    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d43bd84_1533_4057_b4ca_fcbf013f505b.slice/crio-b1a8ac028317381323f113d86a7b33528ca868b9c76301c55527bcbc2eb943af WatchSource:0}: Error finding container b1a8ac028317381323f113d86a7b33528ca868b9c76301c55527bcbc2eb943af: Status 404 returned error can't find the container with id b1a8ac028317381323f113d86a7b33528ca868b9c76301c55527bcbc2eb943af
Sep 08 00:27:27.746498 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.746479    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:27:27.885695 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:27.885655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" event={"ID":"3d43bd84-1533-4057-b4ca-fcbf013f505b","Type":"ContainerStarted","Data":"b1a8ac028317381323f113d86a7b33528ca868b9c76301c55527bcbc2eb943af"}
Sep 08 00:27:28.891902 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:28.891860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" event={"ID":"3d43bd84-1533-4057-b4ca-fcbf013f505b","Type":"ContainerStarted","Data":"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"}
Sep 08 00:27:28.892321 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:28.892099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:28.908042 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:28.907995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" podStartSLOduration=1.403607574 podStartE2EDuration="1.907982499s" totalImagesPullingTime="504.374925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:27:28.905907678 +0000 UTC m=+1481.078172561" watchObservedRunningTime="2026-09-08 00:27:28.907982499 +0000 UTC m=+1481.080247382"
Sep 08 00:27:34.902671 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:34.902640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:27:55.341823 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.341785    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:27:55.345303 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.345283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.347345 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.347321    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5d0042-serving-cert\""
Sep 08 00:27:55.353947 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.353920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:27:55.384862 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.384828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.385040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.384913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.385040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.384950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xns6\" (UniqueName: \"kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.385040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.384981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.486266 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.486219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xns6\" (UniqueName: \"kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.486266 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.486272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.486578 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.486354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.486578 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.486422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.486578 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:27:55.486493    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-serving-cert: secret "maas-api-e2e-api-b-5d0042-serving-cert" not found
Sep 08 00:27:55.486755 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:27:55.486584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls podName:37ee84d5-88de-4880-8b39-90c70d89a1f3 nodeName:}" failed. No retries permitted until 2026-09-08 00:27:55.986562952 +0000 UTC m=+1508.158827814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls") pod "maas-api-e2e-api-b-5d0042-f74859b56-dd448" (UID: "37ee84d5-88de-4880-8b39-90c70d89a1f3") : secret "maas-api-e2e-api-b-5d0042-serving-cert" not found
Sep 08 00:27:55.486933 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.486912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.489096 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.489072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.494674 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.494656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xns6\" (UniqueName: \"kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.992217 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.992177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:55.994820 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:55.994792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") pod \"maas-api-e2e-api-b-5d0042-f74859b56-dd448\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:56.256747 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:56.256643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:56.526635 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:56.526607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:27:56.528542 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:27:56.528497    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ee84d5_88de_4880_8b39_90c70d89a1f3.slice/crio-3cec3e39473462ec407dfc1489a7d990172cbb5091b4d48972940e2759117e29 WatchSource:0}: Error finding container 3cec3e39473462ec407dfc1489a7d990172cbb5091b4d48972940e2759117e29: Status 404 returned error can't find the container with id 3cec3e39473462ec407dfc1489a7d990172cbb5091b4d48972940e2759117e29
Sep 08 00:27:57.025392 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:57.025353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" event={"ID":"37ee84d5-88de-4880-8b39-90c70d89a1f3","Type":"ContainerStarted","Data":"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"}
Sep 08 00:27:57.025610 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:57.025407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" event={"ID":"37ee84d5-88de-4880-8b39-90c70d89a1f3","Type":"ContainerStarted","Data":"3cec3e39473462ec407dfc1489a7d990172cbb5091b4d48972940e2759117e29"}
Sep 08 00:27:57.025610 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:57.025428    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:27:57.041946 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:27:57.041890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" podStartSLOduration=1.674718406 podStartE2EDuration="2.041871147s" totalImagesPullingTime="367.152741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:27:57.038957179 +0000 UTC m=+1509.211222063" watchObservedRunningTime="2026-09-08 00:27:57.041871147 +0000 UTC m=+1509.214136051"
Sep 08 00:28:03.036099 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:03.036062    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:28:37.172701 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.172658    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:28:37.173247 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.172869    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" podUID="3d43bd84-1533-4057-b4ca-fcbf013f505b" containerName="maas-api" containerID="cri-o://7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2" gracePeriod=30
Sep 08 00:28:37.430196 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.430136    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:28:37.473270 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.473240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls\") pod \"3d43bd84-1533-4057-b4ca-fcbf013f505b\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") "
Sep 08 00:28:37.473450 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.473362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs\") pod \"3d43bd84-1533-4057-b4ca-fcbf013f505b\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") "
Sep 08 00:28:37.473450 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.473397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca\") pod \"3d43bd84-1533-4057-b4ca-fcbf013f505b\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") "
Sep 08 00:28:37.473450 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.473433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfqbt\" (UniqueName: \"kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt\") pod \"3d43bd84-1533-4057-b4ca-fcbf013f505b\" (UID: \"3d43bd84-1533-4057-b4ca-fcbf013f505b\") "
Sep 08 00:28:37.473765 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.473739    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d43bd84-1533-4057-b4ca-fcbf013f505b" (UID: "3d43bd84-1533-4057-b4ca-fcbf013f505b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:28:37.475325 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.475300    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d43bd84-1533-4057-b4ca-fcbf013f505b" (UID: "3d43bd84-1533-4057-b4ca-fcbf013f505b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:28:37.475454 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.475432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3d43bd84-1533-4057-b4ca-fcbf013f505b" (UID: "3d43bd84-1533-4057-b4ca-fcbf013f505b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:28:37.475519 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.475437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt" (OuterVolumeSpecName: "kube-api-access-zfqbt") pod "3d43bd84-1533-4057-b4ca-fcbf013f505b" (UID: "3d43bd84-1533-4057-b4ca-fcbf013f505b"). InnerVolumeSpecName "kube-api-access-zfqbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:28:37.574634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.574597    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:28:37.574634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.574628    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d43bd84-1533-4057-b4ca-fcbf013f505b-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:28:37.574634 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.574639    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd84-1533-4057-b4ca-fcbf013f505b-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:28:37.574867 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:37.574648    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqbt\" (UniqueName: \"kubernetes.io/projected/3d43bd84-1533-4057-b4ca-fcbf013f505b-kube-api-access-zfqbt\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:28:38.217988 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.217948    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d43bd84-1533-4057-b4ca-fcbf013f505b" containerID="7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2" exitCode=0
Sep 08 00:28:38.218395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.217993    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"
Sep 08 00:28:38.218395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.218032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" event={"ID":"3d43bd84-1533-4057-b4ca-fcbf013f505b","Type":"ContainerDied","Data":"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"}
Sep 08 00:28:38.218395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.218086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp" event={"ID":"3d43bd84-1533-4057-b4ca-fcbf013f505b","Type":"ContainerDied","Data":"b1a8ac028317381323f113d86a7b33528ca868b9c76301c55527bcbc2eb943af"}
Sep 08 00:28:38.218395 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.218114    2580 scope.go:117] "RemoveContainer" containerID="7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"
Sep 08 00:28:38.227729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.227712    2580 scope.go:117] "RemoveContainer" containerID="7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"
Sep 08 00:28:38.227989 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:28:38.227970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2\": container with ID starting with 7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2 not found: ID does not exist" containerID="7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"
Sep 08 00:28:38.228050 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.228001    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2"} err="failed to get container status \"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2\": rpc error: code = NotFound desc = could not find container \"7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2\": container with ID starting with 7dfcf02eae678ccdbcf61a414fcb55638602d59178b6936dee4c661291352bf2 not found: ID does not exist"
Sep 08 00:28:38.241221 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.241198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:28:38.244876 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.244854    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-93c8ee-999f6b4c5-pvldp"]
Sep 08 00:28:38.330177 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:38.330150    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d43bd84-1533-4057-b4ca-fcbf013f505b" path="/var/lib/kubelet/pods/3d43bd84-1533-4057-b4ca-fcbf013f505b/volumes"
Sep 08 00:28:59.761853 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:59.761817    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:28:59.764500 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:28:59.762082    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" podUID="37ee84d5-88de-4880-8b39-90c70d89a1f3" containerName="maas-api" containerID="cri-o://c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708" gracePeriod=30
Sep 08 00:29:00.002366 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.002337    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:29:00.085435 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.085360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") pod \"37ee84d5-88de-4880-8b39-90c70d89a1f3\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") "
Sep 08 00:29:00.085435 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.085391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca\") pod \"37ee84d5-88de-4880-8b39-90c70d89a1f3\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") "
Sep 08 00:29:00.085701 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.085482    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xns6\" (UniqueName: \"kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6\") pod \"37ee84d5-88de-4880-8b39-90c70d89a1f3\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") "
Sep 08 00:29:00.085701 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.085518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs\") pod \"37ee84d5-88de-4880-8b39-90c70d89a1f3\" (UID: \"37ee84d5-88de-4880-8b39-90c70d89a1f3\") "
Sep 08 00:29:00.085862 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.085835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37ee84d5-88de-4880-8b39-90c70d89a1f3" (UID: "37ee84d5-88de-4880-8b39-90c70d89a1f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:29:00.087639 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.087606    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37ee84d5-88de-4880-8b39-90c70d89a1f3" (UID: "37ee84d5-88de-4880-8b39-90c70d89a1f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:00.087752 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.087713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37ee84d5-88de-4880-8b39-90c70d89a1f3" (UID: "37ee84d5-88de-4880-8b39-90c70d89a1f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:00.087814 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.087746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6" (OuterVolumeSpecName: "kube-api-access-7xns6") pod "37ee84d5-88de-4880-8b39-90c70d89a1f3" (UID: "37ee84d5-88de-4880-8b39-90c70d89a1f3"). InnerVolumeSpecName "kube-api-access-7xns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:29:00.186868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.186830    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xns6\" (UniqueName: \"kubernetes.io/projected/37ee84d5-88de-4880-8b39-90c70d89a1f3-kube-api-access-7xns6\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:00.186868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.186861    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:00.186868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.186874    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37ee84d5-88de-4880-8b39-90c70d89a1f3-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:00.187098 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.186885    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ee84d5-88de-4880-8b39-90c70d89a1f3-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:00.319595 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.319558    2580 generic.go:356] "Generic (PLEG): container finished" podID="37ee84d5-88de-4880-8b39-90c70d89a1f3" containerID="c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708" exitCode=0
Sep 08 00:29:00.319762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.319600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"
Sep 08 00:29:00.319762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.319613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" event={"ID":"37ee84d5-88de-4880-8b39-90c70d89a1f3","Type":"ContainerDied","Data":"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"}
Sep 08 00:29:00.319762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.319655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448" event={"ID":"37ee84d5-88de-4880-8b39-90c70d89a1f3","Type":"ContainerDied","Data":"3cec3e39473462ec407dfc1489a7d990172cbb5091b4d48972940e2759117e29"}
Sep 08 00:29:00.319762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.319676    2580 scope.go:117] "RemoveContainer" containerID="c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"
Sep 08 00:29:00.329762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.329743    2580 scope.go:117] "RemoveContainer" containerID="c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"
Sep 08 00:29:00.330001 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:29:00.329978    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708\": container with ID starting with c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708 not found: ID does not exist" containerID="c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"
Sep 08 00:29:00.330068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.330010    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708"} err="failed to get container status \"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708\": rpc error: code = NotFound desc = could not find container \"c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708\": container with ID starting with c491b4696a8b5d50ff3fed1a747d07735f1b9035f8be068e25b38a218b2ce708 not found: ID does not exist"
Sep 08 00:29:00.339847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.339791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:29:00.345322 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:00.345302    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d0042-f74859b56-dd448"]
Sep 08 00:29:02.331120 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:02.331083    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ee84d5-88de-4880-8b39-90c70d89a1f3" path="/var/lib/kubelet/pods/37ee84d5-88de-4880-8b39-90c70d89a1f3/volumes"
Sep 08 00:29:17.034091 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.034055    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:29:17.034667 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.034247    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" podUID="24323acc-0617-4f94-ba0c-148c9ac62b99" containerName="maas-api" containerID="cri-o://48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1" gracePeriod=30
Sep 08 00:29:17.092158 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.092121    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:29:17.092411 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.092382    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" podUID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" containerName="payload-pre-processing" containerID="cri-o://bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b" gracePeriod=30
Sep 08 00:29:17.331346 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.331308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:29:17.386563 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.386515    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:29:17.400752 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.400721    2580 generic.go:356] "Generic (PLEG): container finished" podID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" containerID="bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b" exitCode=0
Sep 08 00:29:17.400913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.400774    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"
Sep 08 00:29:17.400913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.400805    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" event={"ID":"51e467e0-e7ed-40fb-b801-d5a386c93cfe","Type":"ContainerDied","Data":"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"}
Sep 08 00:29:17.400913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.400858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd" event={"ID":"51e467e0-e7ed-40fb-b801-d5a386c93cfe","Type":"ContainerDied","Data":"d01cba60719ae9198c864c5a2c9b28b9e36c471625f525875e8261b93b5e4bf8"}
Sep 08 00:29:17.400913 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.400888    2580 scope.go:117] "RemoveContainer" containerID="bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"
Sep 08 00:29:17.405618 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.403212    2580 generic.go:356] "Generic (PLEG): container finished" podID="24323acc-0617-4f94-ba0c-148c9ac62b99" containerID="48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1" exitCode=0
Sep 08 00:29:17.405618 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.403254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" event={"ID":"24323acc-0617-4f94-ba0c-148c9ac62b99","Type":"ContainerDied","Data":"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"}
Sep 08 00:29:17.405618 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.403294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv" event={"ID":"24323acc-0617-4f94-ba0c-148c9ac62b99","Type":"ContainerDied","Data":"b7b81e3befc9f8f2095d0971d319a455af955341f15b63d61931f15c85c07357"}
Sep 08 00:29:17.405618 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.403361    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"
Sep 08 00:29:17.414206 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.414183    2580 scope.go:117] "RemoveContainer" containerID="bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"
Sep 08 00:29:17.414583 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:29:17.414562    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b\": container with ID starting with bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b not found: ID does not exist" containerID="bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"
Sep 08 00:29:17.414678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.414594    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b"} err="failed to get container status \"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b\": rpc error: code = NotFound desc = could not find container \"bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b\": container with ID starting with bfb0fd5e1c5e9a8618ce6278d864ea85c0c9db4ac4e55947d66527baa5b5ef7b not found: ID does not exist"
Sep 08 00:29:17.414678 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.414611    2580 scope.go:117] "RemoveContainer" containerID="48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"
Sep 08 00:29:17.424018 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.423996    2580 scope.go:117] "RemoveContainer" containerID="48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"
Sep 08 00:29:17.424301 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:29:17.424282    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1\": container with ID starting with 48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1 not found: ID does not exist" containerID="48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"
Sep 08 00:29:17.424368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.424309    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1"} err="failed to get container status \"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1\": rpc error: code = NotFound desc = could not find container \"48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1\": container with ID starting with 48d251f0e3988e60fab59740859d79acdfbeb09679ef2dfaaf888d572cdcaea1 not found: ID does not exist"
Sep 08 00:29:17.432769 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") pod \"24323acc-0617-4f94-ba0c-148c9ac62b99\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") "
Sep 08 00:29:17.432875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74vrk\" (UniqueName: \"kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk\") pod \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") "
Sep 08 00:29:17.432875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh2qp\" (UniqueName: \"kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp\") pod \"24323acc-0617-4f94-ba0c-148c9ac62b99\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") "
Sep 08 00:29:17.433002 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume\") pod \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\" (UID: \"51e467e0-e7ed-40fb-b801-d5a386c93cfe\") "
Sep 08 00:29:17.433002 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca\") pod \"24323acc-0617-4f94-ba0c-148c9ac62b99\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") "
Sep 08 00:29:17.433002 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.432994    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs\") pod \"24323acc-0617-4f94-ba0c-148c9ac62b99\" (UID: \"24323acc-0617-4f94-ba0c-148c9ac62b99\") "
Sep 08 00:29:17.433495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.433443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51e467e0-e7ed-40fb-b801-d5a386c93cfe" (UID: "51e467e0-e7ed-40fb-b801-d5a386c93cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:29:17.433495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.433488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24323acc-0617-4f94-ba0c-148c9ac62b99" (UID: "24323acc-0617-4f94-ba0c-148c9ac62b99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:29:17.435360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.435339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24323acc-0617-4f94-ba0c-148c9ac62b99" (UID: "24323acc-0617-4f94-ba0c-148c9ac62b99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:17.435444 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.435392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk" (OuterVolumeSpecName: "kube-api-access-74vrk") pod "51e467e0-e7ed-40fb-b801-d5a386c93cfe" (UID: "51e467e0-e7ed-40fb-b801-d5a386c93cfe"). InnerVolumeSpecName "kube-api-access-74vrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:29:17.435506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.435446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp" (OuterVolumeSpecName: "kube-api-access-zh2qp") pod "24323acc-0617-4f94-ba0c-148c9ac62b99" (UID: "24323acc-0617-4f94-ba0c-148c9ac62b99"). InnerVolumeSpecName "kube-api-access-zh2qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:29:17.435506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.435480    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24323acc-0617-4f94-ba0c-148c9ac62b99" (UID: "24323acc-0617-4f94-ba0c-148c9ac62b99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:17.534160 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534121    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2qp\" (UniqueName: \"kubernetes.io/projected/24323acc-0617-4f94-ba0c-148c9ac62b99-kube-api-access-zh2qp\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.534160 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534153    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e467e0-e7ed-40fb-b801-d5a386c93cfe-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.534160 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534162    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24323acc-0617-4f94-ba0c-148c9ac62b99-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.534160 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534172    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.534431 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534181    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24323acc-0617-4f94-ba0c-148c9ac62b99-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.534431 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.534189    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vrk\" (UniqueName: \"kubernetes.io/projected/51e467e0-e7ed-40fb-b801-d5a386c93cfe-kube-api-access-74vrk\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:17.723040 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.723011    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:29:17.726184 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.726158    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c6a703-7ddb567c4b-jc8nd"]
Sep 08 00:29:17.735403 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.735372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:29:17.738942 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:17.738915    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c6a703-7bc96d59fc-mrbrv"]
Sep 08 00:29:18.330062 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:18.329979    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24323acc-0617-4f94-ba0c-148c9ac62b99" path="/var/lib/kubelet/pods/24323acc-0617-4f94-ba0c-148c9ac62b99/volumes"
Sep 08 00:29:18.330397 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:18.330334    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" path="/var/lib/kubelet/pods/51e467e0-e7ed-40fb-b801-d5a386c93cfe/volumes"
Sep 08 00:29:34.240658 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.240599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:29:34.241171 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.240888    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" podUID="bca2753b-08b3-4b94-aa47-44da37b88ae3" containerName="maas-api" containerID="cri-o://a7fe71cc6b6a96ac32d683c2dd391548fd8583d43596343969378d555e5335de" gracePeriod=30
Sep 08 00:29:34.485884 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.485845    2580 generic.go:356] "Generic (PLEG): container finished" podID="bca2753b-08b3-4b94-aa47-44da37b88ae3" containerID="a7fe71cc6b6a96ac32d683c2dd391548fd8583d43596343969378d555e5335de" exitCode=0
Sep 08 00:29:34.486006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.485915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" event={"ID":"bca2753b-08b3-4b94-aa47-44da37b88ae3","Type":"ContainerDied","Data":"a7fe71cc6b6a96ac32d683c2dd391548fd8583d43596343969378d555e5335de"}
Sep 08 00:29:34.486006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.485968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k" event={"ID":"bca2753b-08b3-4b94-aa47-44da37b88ae3","Type":"ContainerDied","Data":"cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0"}
Sep 08 00:29:34.486006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.485984    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfebaa6e8a77a24b35f974c9bd4cc519baada5d8b64e61f4474410d575a593e0"
Sep 08 00:29:34.494952 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.494892    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:29:34.592077 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.592044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs\") pod \"bca2753b-08b3-4b94-aa47-44da37b88ae3\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") "
Sep 08 00:29:34.592266 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.592102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgrkl\" (UniqueName: \"kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl\") pod \"bca2753b-08b3-4b94-aa47-44da37b88ae3\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") "
Sep 08 00:29:34.592266 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.592150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls\") pod \"bca2753b-08b3-4b94-aa47-44da37b88ae3\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") "
Sep 08 00:29:34.592266 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.592235    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca\") pod \"bca2753b-08b3-4b94-aa47-44da37b88ae3\" (UID: \"bca2753b-08b3-4b94-aa47-44da37b88ae3\") "
Sep 08 00:29:34.592651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.592620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bca2753b-08b3-4b94-aa47-44da37b88ae3" (UID: "bca2753b-08b3-4b94-aa47-44da37b88ae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:29:34.594215 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.594183    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bca2753b-08b3-4b94-aa47-44da37b88ae3" (UID: "bca2753b-08b3-4b94-aa47-44da37b88ae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:34.594215 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.594203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bca2753b-08b3-4b94-aa47-44da37b88ae3" (UID: "bca2753b-08b3-4b94-aa47-44da37b88ae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:29:34.594341 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.594206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl" (OuterVolumeSpecName: "kube-api-access-fgrkl") pod "bca2753b-08b3-4b94-aa47-44da37b88ae3" (UID: "bca2753b-08b3-4b94-aa47-44da37b88ae3"). InnerVolumeSpecName "kube-api-access-fgrkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:29:34.692954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.692919    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca2753b-08b3-4b94-aa47-44da37b88ae3-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:34.692954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.692948    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:34.692954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.692957    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrkl\" (UniqueName: \"kubernetes.io/projected/bca2753b-08b3-4b94-aa47-44da37b88ae3-kube-api-access-fgrkl\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:34.693177 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:34.692966    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca2753b-08b3-4b94-aa47-44da37b88ae3-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:29:35.490572 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:35.490519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"
Sep 08 00:29:35.510086 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:35.510052    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:29:35.514061 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:35.514036    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e1e91c-658d8846d6-7h85k"]
Sep 08 00:29:36.330426 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:36.330383    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca2753b-08b3-4b94-aa47-44da37b88ae3" path="/var/lib/kubelet/pods/bca2753b-08b3-4b94-aa47-44da37b88ae3/volumes"
Sep 08 00:29:48.438153 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:29:48.438120    2580 scope.go:117] "RemoveContainer" containerID="a7fe71cc6b6a96ac32d683c2dd391548fd8583d43596343969378d555e5335de"
Sep 08 00:30:04.235022 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.234979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:30:04.235571 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235551    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24323acc-0617-4f94-ba0c-148c9ac62b99" containerName="maas-api"
Sep 08 00:30:04.235623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235574    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24323acc-0617-4f94-ba0c-148c9ac62b99" containerName="maas-api"
Sep 08 00:30:04.235623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235586    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d43bd84-1533-4057-b4ca-fcbf013f505b" containerName="maas-api"
Sep 08 00:30:04.235623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235594    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d43bd84-1533-4057-b4ca-fcbf013f505b" containerName="maas-api"
Sep 08 00:30:04.235623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235610    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" containerName="payload-pre-processing"
Sep 08 00:30:04.235623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235619    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" containerName="payload-pre-processing"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235635    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ee84d5-88de-4880-8b39-90c70d89a1f3" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235644    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ee84d5-88de-4880-8b39-90c70d89a1f3" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235664    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca2753b-08b3-4b94-aa47-44da37b88ae3" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235673    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca2753b-08b3-4b94-aa47-44da37b88ae3" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235753    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d43bd84-1533-4057-b4ca-fcbf013f505b" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235765    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24323acc-0617-4f94-ba0c-148c9ac62b99" containerName="maas-api"
Sep 08 00:30:04.235785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235777    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca2753b-08b3-4b94-aa47-44da37b88ae3" containerName="maas-api"
Sep 08 00:30:04.235997 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235790    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e467e0-e7ed-40fb-b801-d5a386c93cfe" containerName="payload-pre-processing"
Sep 08 00:30:04.235997 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.235799    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ee84d5-88de-4880-8b39-90c70d89a1f3" containerName="maas-api"
Sep 08 00:30:04.239236 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.239218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.241564 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.241514    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8098ed-serving-cert\""
Sep 08 00:30:04.247623 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.247556    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:30:04.260158 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.260131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xq2k\" (UniqueName: \"kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.260283 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.260228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.260283 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.260253    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.260283 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.260270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.361753 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.361720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xq2k\" (UniqueName: \"kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.361936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.361822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.361936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.361855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.361936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.361881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.362129 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:30:04.361969    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-serving-cert: secret "maas-api-e2e-ipp-a-8098ed-serving-cert" not found
Sep 08 00:30:04.362129 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:30:04.362040    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls podName:847d4e5a-ad2f-4d9e-9996-37f67eb64fbb nodeName:}" failed. No retries permitted until 2026-09-08 00:30:04.862020501 +0000 UTC m=+1637.034285377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls") pod "maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" (UID: "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb") : secret "maas-api-e2e-ipp-a-8098ed-serving-cert" not found
Sep 08 00:30:04.362454 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.362432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.364273 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.364249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.371679 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.371658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xq2k\" (UniqueName: \"kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.866123 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.866087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:04.868488 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:04.868463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:05.151719 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:05.151680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:05.421652 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:05.421627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:30:05.424108 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:30:05.424083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847d4e5a_ad2f_4d9e_9996_37f67eb64fbb.slice/crio-2d6d0f1c24331d5552261223f79987a6fa2bac9a17372358bba8f1d50cd89b1b WatchSource:0}: Error finding container 2d6d0f1c24331d5552261223f79987a6fa2bac9a17372358bba8f1d50cd89b1b: Status 404 returned error can't find the container with id 2d6d0f1c24331d5552261223f79987a6fa2bac9a17372358bba8f1d50cd89b1b
Sep 08 00:30:05.626732 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:05.626699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" event={"ID":"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb","Type":"ContainerStarted","Data":"2d6d0f1c24331d5552261223f79987a6fa2bac9a17372358bba8f1d50cd89b1b"}
Sep 08 00:30:06.632679 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:06.632593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" event={"ID":"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb","Type":"ContainerStarted","Data":"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"}
Sep 08 00:30:06.633085 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:06.632718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:06.649113 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:06.649063    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" podStartSLOduration=1.89175362 podStartE2EDuration="2.649050081s" totalImagesPullingTime="757.296461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:30:06.646419306 +0000 UTC m=+1638.818684187" watchObservedRunningTime="2026-09-08 00:30:06.649050081 +0000 UTC m=+1638.821314966"
Sep 08 00:30:12.642002 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:12.641965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:30:33.484089 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.484058    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:30:33.487473 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.487451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.489639 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.489612    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4d2f00-serving-cert\""
Sep 08 00:30:33.498969 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.498943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:30:33.622889 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.622853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.623058 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.622899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.623058 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.622973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.623058 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.622995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6428m\" (UniqueName: \"kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.724055 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.724015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.724249 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.724070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.724249 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.724155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.724249 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.724189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6428m\" (UniqueName: \"kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.724827 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.724799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.726563 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.726502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.726563 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.726522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.742674 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.741767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6428m\" (UniqueName: \"kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m\") pod \"maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:33.799351 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:33.799319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:34.071591 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.071485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:30:34.074308 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:30:34.074274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487de525_0f58_4d2d_964f_42b8355fab8d.slice/crio-6b4766d2316b72ee54977f61902186fcf8c3d71149648cf4340f25d19275686a WatchSource:0}: Error finding container 6b4766d2316b72ee54977f61902186fcf8c3d71149648cf4340f25d19275686a: Status 404 returned error can't find the container with id 6b4766d2316b72ee54977f61902186fcf8c3d71149648cf4340f25d19275686a
Sep 08 00:30:34.132512 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.132479    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:30:34.137057 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.137039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.140596 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.140580    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4d2f00-dockercfg-g8m4k\""
Sep 08 00:30:34.140864 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.140843    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4d2f00\""
Sep 08 00:30:34.152758 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.152736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:30:34.228960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.228727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.228960 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.228877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2hg\" (UniqueName: \"kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.329697 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.329605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.329697 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.329686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2hg\" (UniqueName: \"kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.330286 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.330264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.336995 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.336975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2hg\" (UniqueName: \"kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg\") pod \"payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.447317 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.447286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:34.715727 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.715703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:30:34.717794 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:30:34.717768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c1e1ac_9c30_47d0_9100_ee2f45ad2f92.slice/crio-e7ce23e603fef20017ce8de52cfbdf0d5d1383c42c30fe20ebe60f9895ef7f54 WatchSource:0}: Error finding container e7ce23e603fef20017ce8de52cfbdf0d5d1383c42c30fe20ebe60f9895ef7f54: Status 404 returned error can't find the container with id e7ce23e603fef20017ce8de52cfbdf0d5d1383c42c30fe20ebe60f9895ef7f54
Sep 08 00:30:34.758414 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.758385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" event={"ID":"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92","Type":"ContainerStarted","Data":"e7ce23e603fef20017ce8de52cfbdf0d5d1383c42c30fe20ebe60f9895ef7f54"}
Sep 08 00:30:34.760122 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.760100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" event={"ID":"487de525-0f58-4d2d-964f-42b8355fab8d","Type":"ContainerStarted","Data":"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"}
Sep 08 00:30:34.760235 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.760147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" event={"ID":"487de525-0f58-4d2d-964f-42b8355fab8d","Type":"ContainerStarted","Data":"6b4766d2316b72ee54977f61902186fcf8c3d71149648cf4340f25d19275686a"}
Sep 08 00:30:34.760306 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.760226    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:34.780353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:34.780301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" podStartSLOduration=1.383471952 podStartE2EDuration="1.78028039s" totalImagesPullingTime="396.808438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:30:34.77791203 +0000 UTC m=+1666.950176916" watchObservedRunningTime="2026-09-08 00:30:34.78028039 +0000 UTC m=+1666.952545272"
Sep 08 00:30:35.766779 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:35.766736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" event={"ID":"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92","Type":"ContainerStarted","Data":"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"}
Sep 08 00:30:35.767197 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:35.766982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:30:35.782137 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:35.782089    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" podStartSLOduration=1.7820728209999999 podStartE2EDuration="1.782072821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:30:35.780760156 +0000 UTC m=+1667.953025043" watchObservedRunningTime="2026-09-08 00:30:35.782072821 +0000 UTC m=+1667.954337707"
Sep 08 00:30:40.772650 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:40.772616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:30:46.774375 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:30:46.774339    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:31:42.049123 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.049084    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:31:42.053433 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.053411    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.055632 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.055610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-3d0f11-serving-cert\""
Sep 08 00:31:42.060639 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.060612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:31:42.151705 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.151669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.151914 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.151785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.151914 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.151844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwvp\" (UniqueName: \"kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.151914 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.151893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.252707 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.252669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.252875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.252739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.252875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.252764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwvp\" (UniqueName: \"kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.252875 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.252792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.252979 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:31:42.252912    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-serving-cert: secret "maas-api-e2e-ipp-cleanup-3d0f11-serving-cert" not found
Sep 08 00:31:42.253016 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:31:42.252994    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls podName:dcbd4f83-509d-48bd-af46-47a6a8da3e25 nodeName:}" failed. No retries permitted until 2026-09-08 00:31:42.752971062 +0000 UTC m=+1734.925235928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" (UID: "dcbd4f83-509d-48bd-af46-47a6a8da3e25") : secret "maas-api-e2e-ipp-cleanup-3d0f11-serving-cert" not found
Sep 08 00:31:42.253427 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.253406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.255137 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.255116    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.263806 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.263786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwvp\" (UniqueName: \"kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.757553 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.757486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.760001 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.759979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:42.829215 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.829176    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:31:42.833459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.833438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:42.836481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.836456    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3d0f11\""
Sep 08 00:31:42.836599 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.836464    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3d0f11-dockercfg-6mjn5\""
Sep 08 00:31:42.843630 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.843595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:31:42.959405 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.959368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:42.959599 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.959443    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fhb\" (UniqueName: \"kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:42.966245 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:42.966224    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:43.060297 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.060257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:43.060680 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.060419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fhb\" (UniqueName: \"kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:43.061068 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.061047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:43.068963 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.068939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fhb\" (UniqueName: \"kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:43.145640 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.145608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:43.241445 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.241419    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:31:43.243270 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:31:43.243244    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbd4f83_509d_48bd_af46_47a6a8da3e25.slice/crio-ceee62f530e3dac08c289b76e8dbab8f1da27aab5bd37592b802634690a3891c WatchSource:0}: Error finding container ceee62f530e3dac08c289b76e8dbab8f1da27aab5bd37592b802634690a3891c: Status 404 returned error can't find the container with id ceee62f530e3dac08c289b76e8dbab8f1da27aab5bd37592b802634690a3891c
Sep 08 00:31:43.419651 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:43.419625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:31:43.421294 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:31:43.421267    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457e56bb_79a9_412a_928d_0a695bcae88d.slice/crio-8728d528f210f0a4a333ece08eb83b48f5b31905511d684aba8fcddd3726968a WatchSource:0}: Error finding container 8728d528f210f0a4a333ece08eb83b48f5b31905511d684aba8fcddd3726968a: Status 404 returned error can't find the container with id 8728d528f210f0a4a333ece08eb83b48f5b31905511d684aba8fcddd3726968a
Sep 08 00:31:44.103941 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.103859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" event={"ID":"457e56bb-79a9-412a-928d-0a695bcae88d","Type":"ContainerStarted","Data":"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"}
Sep 08 00:31:44.103941 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.103934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" event={"ID":"457e56bb-79a9-412a-928d-0a695bcae88d","Type":"ContainerStarted","Data":"8728d528f210f0a4a333ece08eb83b48f5b31905511d684aba8fcddd3726968a"}
Sep 08 00:31:44.104443 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.103974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:31:44.105562 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.105519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" event={"ID":"dcbd4f83-509d-48bd-af46-47a6a8da3e25","Type":"ContainerStarted","Data":"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"}
Sep 08 00:31:44.105683 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.105576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" event={"ID":"dcbd4f83-509d-48bd-af46-47a6a8da3e25","Type":"ContainerStarted","Data":"ceee62f530e3dac08c289b76e8dbab8f1da27aab5bd37592b802634690a3891c"}
Sep 08 00:31:44.105752 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.105680    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:44.120609 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.120564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" podStartSLOduration=2.120549214 podStartE2EDuration="2.120549214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:31:44.119736903 +0000 UTC m=+1736.292001788" watchObservedRunningTime="2026-09-08 00:31:44.120549214 +0000 UTC m=+1736.292814096"
Sep 08 00:31:44.137581 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:44.137523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" podStartSLOduration=1.713005532 podStartE2EDuration="2.137511077s" totalImagesPullingTime="424.505545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:31:44.136252163 +0000 UTC m=+1736.308517051" watchObservedRunningTime="2026-09-08 00:31:44.137511077 +0000 UTC m=+1736.309775960"
Sep 08 00:31:50.115858 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:50.115827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:31:55.112706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:31:55.112667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:32:12.790488 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:12.790450    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:32:12.791081 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:12.790704    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" podUID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" containerName="maas-api" containerID="cri-o://0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87" gracePeriod=30
Sep 08 00:32:12.855767 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:12.855727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:32:12.856021 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:12.855992    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" podUID="457e56bb-79a9-412a-928d-0a695bcae88d" containerName="payload-pre-processing" containerID="cri-o://6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae" gracePeriod=30
Sep 08 00:32:13.097768 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.097741    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:32:13.131893 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.131864    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:32:13.151345 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151310    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwvp\" (UniqueName: \"kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp\") pod \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") "
Sep 08 00:32:13.151506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca\") pod \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") "
Sep 08 00:32:13.151506 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs\") pod \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") "
Sep 08 00:32:13.151654 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151511    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") pod \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\" (UID: \"dcbd4f83-509d-48bd-af46-47a6a8da3e25\") "
Sep 08 00:32:13.151654 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9fhb\" (UniqueName: \"kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb\") pod \"457e56bb-79a9-412a-928d-0a695bcae88d\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") "
Sep 08 00:32:13.151654 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.151630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume\") pod \"457e56bb-79a9-412a-928d-0a695bcae88d\" (UID: \"457e56bb-79a9-412a-928d-0a695bcae88d\") "
Sep 08 00:32:13.152436 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.152239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "457e56bb-79a9-412a-928d-0a695bcae88d" (UID: "457e56bb-79a9-412a-928d-0a695bcae88d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:32:13.152789 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.152738    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dcbd4f83-509d-48bd-af46-47a6a8da3e25" (UID: "dcbd4f83-509d-48bd-af46-47a6a8da3e25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:32:13.154433 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.154406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dcbd4f83-509d-48bd-af46-47a6a8da3e25" (UID: "dcbd4f83-509d-48bd-af46-47a6a8da3e25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:32:13.154631 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.154608    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb" (OuterVolumeSpecName: "kube-api-access-k9fhb") pod "457e56bb-79a9-412a-928d-0a695bcae88d" (UID: "457e56bb-79a9-412a-928d-0a695bcae88d"). InnerVolumeSpecName "kube-api-access-k9fhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:32:13.154706 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.154675    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dcbd4f83-509d-48bd-af46-47a6a8da3e25" (UID: "dcbd4f83-509d-48bd-af46-47a6a8da3e25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:32:13.156063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.156040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp" (OuterVolumeSpecName: "kube-api-access-plwvp") pod "dcbd4f83-509d-48bd-af46-47a6a8da3e25" (UID: "dcbd4f83-509d-48bd-af46-47a6a8da3e25"). InnerVolumeSpecName "kube-api-access-plwvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:32:13.243192 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.243147    2580 generic.go:356] "Generic (PLEG): container finished" podID="457e56bb-79a9-412a-928d-0a695bcae88d" containerID="6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae" exitCode=0
Sep 08 00:32:13.243353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.243204    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"
Sep 08 00:32:13.243353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.243227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" event={"ID":"457e56bb-79a9-412a-928d-0a695bcae88d","Type":"ContainerDied","Data":"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"}
Sep 08 00:32:13.243353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.243286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh" event={"ID":"457e56bb-79a9-412a-928d-0a695bcae88d","Type":"ContainerDied","Data":"8728d528f210f0a4a333ece08eb83b48f5b31905511d684aba8fcddd3726968a"}
Sep 08 00:32:13.243353 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.243315    2580 scope.go:117] "RemoveContainer" containerID="6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"
Sep 08 00:32:13.244516 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.244498    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" containerID="0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87" exitCode=0
Sep 08 00:32:13.244615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.244554    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"
Sep 08 00:32:13.244615 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.244570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" event={"ID":"dcbd4f83-509d-48bd-af46-47a6a8da3e25","Type":"ContainerDied","Data":"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"}
Sep 08 00:32:13.244714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.244619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz" event={"ID":"dcbd4f83-509d-48bd-af46-47a6a8da3e25","Type":"ContainerDied","Data":"ceee62f530e3dac08c289b76e8dbab8f1da27aab5bd37592b802634690a3891c"}
Sep 08 00:32:13.252444 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252423    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcbd4f83-509d-48bd-af46-47a6a8da3e25-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.252552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252446    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.252552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252461    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcbd4f83-509d-48bd-af46-47a6a8da3e25-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.252552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252473    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fhb\" (UniqueName: \"kubernetes.io/projected/457e56bb-79a9-412a-928d-0a695bcae88d-kube-api-access-k9fhb\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.252552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252485    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457e56bb-79a9-412a-928d-0a695bcae88d-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.252552 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.252499    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwvp\" (UniqueName: \"kubernetes.io/projected/dcbd4f83-509d-48bd-af46-47a6a8da3e25-kube-api-access-plwvp\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:13.253567 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.253551    2580 scope.go:117] "RemoveContainer" containerID="6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"
Sep 08 00:32:13.253905 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:32:13.253886    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae\": container with ID starting with 6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae not found: ID does not exist" containerID="6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"
Sep 08 00:32:13.253965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.253914    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae"} err="failed to get container status \"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae\": rpc error: code = NotFound desc = could not find container \"6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae\": container with ID starting with 6fa5f543fb8cf948da71d97a0cdaaf17885f5ea0f9fe9b3622d4dca4e99e4aae not found: ID does not exist"
Sep 08 00:32:13.253965 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.253932    2580 scope.go:117] "RemoveContainer" containerID="0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"
Sep 08 00:32:13.261790 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.261773    2580 scope.go:117] "RemoveContainer" containerID="0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"
Sep 08 00:32:13.262047 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:32:13.262025    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87\": container with ID starting with 0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87 not found: ID does not exist" containerID="0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"
Sep 08 00:32:13.262103 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.262053    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87"} err="failed to get container status \"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87\": rpc error: code = NotFound desc = could not find container \"0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87\": container with ID starting with 0dc26fbaf51bab061992a0589eb03cc90a38712e204129ed904e8a7e000adb87 not found: ID does not exist"
Sep 08 00:32:13.266777 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.266755    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:32:13.269259 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.269239    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d0f11-6655b7df74-qv2wh"]
Sep 08 00:32:13.280982 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.280960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:32:13.287847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:13.287826    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-3d0f11-7f7f7bf7fc-bcqdz"]
Sep 08 00:32:14.330775 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:14.330744    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457e56bb-79a9-412a-928d-0a695bcae88d" path="/var/lib/kubelet/pods/457e56bb-79a9-412a-928d-0a695bcae88d/volumes"
Sep 08 00:32:14.331135 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:14.331118    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" path="/var/lib/kubelet/pods/dcbd4f83-509d-48bd-af46-47a6a8da3e25/volumes"
Sep 08 00:32:38.704588 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:38.704551    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:32:38.704999 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:38.704783    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" podUID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" containerName="maas-api" containerID="cri-o://ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9" gracePeriod=30
Sep 08 00:32:38.958655 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:38.958593    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:32:39.097729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.097701    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xq2k\" (UniqueName: \"kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k\") pod \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") "
Sep 08 00:32:39.097729 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.097737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs\") pod \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") "
Sep 08 00:32:39.097992 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.097791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca\") pod \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") "
Sep 08 00:32:39.097992 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.097837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") pod \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\" (UID: \"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb\") "
Sep 08 00:32:39.098223 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.098197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" (UID: "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:32:39.100094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.100061    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" (UID: "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:32:39.100094 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.100365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k" (OuterVolumeSpecName: "kube-api-access-7xq2k") pod "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" (UID: "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb"). InnerVolumeSpecName "kube-api-access-7xq2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:32:39.105893 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.105864    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" (UID: "847d4e5a-ad2f-4d9e-9996-37f67eb64fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:32:39.198755 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.198713    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:39.198954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.198766    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xq2k\" (UniqueName: \"kubernetes.io/projected/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-kube-api-access-7xq2k\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:39.198954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.198777    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:39.198954 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.198786    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:32:39.366397 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.366303    2580 generic.go:356] "Generic (PLEG): container finished" podID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" containerID="ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9" exitCode=0
Sep 08 00:32:39.366397 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.366360    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"
Sep 08 00:32:39.366684 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.366393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" event={"ID":"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb","Type":"ContainerDied","Data":"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"}
Sep 08 00:32:39.366684 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.366457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm" event={"ID":"847d4e5a-ad2f-4d9e-9996-37f67eb64fbb","Type":"ContainerDied","Data":"2d6d0f1c24331d5552261223f79987a6fa2bac9a17372358bba8f1d50cd89b1b"}
Sep 08 00:32:39.366684 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.366487    2580 scope.go:117] "RemoveContainer" containerID="ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"
Sep 08 00:32:39.376378 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.376354    2580 scope.go:117] "RemoveContainer" containerID="ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"
Sep 08 00:32:39.376690 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:32:39.376667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9\": container with ID starting with ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9 not found: ID does not exist" containerID="ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"
Sep 08 00:32:39.376755 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.376700    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9"} err="failed to get container status \"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9\": rpc error: code = NotFound desc = could not find container \"ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9\": container with ID starting with ea83bd6905ade691f9f7c1bf3c4d2ba560c03b3a52ca7aa758e96e5d7a8ee7f9 not found: ID does not exist"
Sep 08 00:32:39.397352 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.397319    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:32:39.403421 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:39.403391    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8098ed-c88d74b55-ghjbm"]
Sep 08 00:32:40.331429 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:32:40.331356    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" path="/var/lib/kubelet/pods/847d4e5a-ad2f-4d9e-9996-37f67eb64fbb/volumes"
Sep 08 00:33:43.181092 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.181057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:33:43.181572 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.181312    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" podUID="487de525-0f58-4d2d-964f-42b8355fab8d" containerName="maas-api" containerID="cri-o://6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed" gracePeriod=30
Sep 08 00:33:43.400179 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.400141    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:33:43.400604 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.400427    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" podUID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" containerName="payload-pre-processing" containerID="cri-o://36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e" gracePeriod=30
Sep 08 00:33:43.525704 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.525679    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:33:43.586087 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6428m\" (UniqueName: \"kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m\") pod \"487de525-0f58-4d2d-964f-42b8355fab8d\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") "
Sep 08 00:33:43.586242 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca\") pod \"487de525-0f58-4d2d-964f-42b8355fab8d\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") "
Sep 08 00:33:43.586242 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586232    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs\") pod \"487de525-0f58-4d2d-964f-42b8355fab8d\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") "
Sep 08 00:33:43.586363 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls\") pod \"487de525-0f58-4d2d-964f-42b8355fab8d\" (UID: \"487de525-0f58-4d2d-964f-42b8355fab8d\") "
Sep 08 00:33:43.586600 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "487de525-0f58-4d2d-964f-42b8355fab8d" (UID: "487de525-0f58-4d2d-964f-42b8355fab8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:33:43.586700 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.586662    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/487de525-0f58-4d2d-964f-42b8355fab8d-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:43.588520 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.588478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "487de525-0f58-4d2d-964f-42b8355fab8d" (UID: "487de525-0f58-4d2d-964f-42b8355fab8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:33:43.588520 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.588478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m" (OuterVolumeSpecName: "kube-api-access-6428m") pod "487de525-0f58-4d2d-964f-42b8355fab8d" (UID: "487de525-0f58-4d2d-964f-42b8355fab8d"). InnerVolumeSpecName "kube-api-access-6428m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:33:43.588742 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.588538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "487de525-0f58-4d2d-964f-42b8355fab8d" (UID: "487de525-0f58-4d2d-964f-42b8355fab8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:33:43.654314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.654290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:33:43.663091 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.663064    2580 generic.go:356] "Generic (PLEG): container finished" podID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" containerID="36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e" exitCode=0
Sep 08 00:33:43.663234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.663106    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"
Sep 08 00:33:43.663234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.663140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" event={"ID":"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92","Type":"ContainerDied","Data":"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"}
Sep 08 00:33:43.663234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.663198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv" event={"ID":"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92","Type":"ContainerDied","Data":"e7ce23e603fef20017ce8de52cfbdf0d5d1383c42c30fe20ebe60f9895ef7f54"}
Sep 08 00:33:43.663234 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.663221    2580 scope.go:117] "RemoveContainer" containerID="36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"
Sep 08 00:33:43.664418 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.664399    2580 generic.go:356] "Generic (PLEG): container finished" podID="487de525-0f58-4d2d-964f-42b8355fab8d" containerID="6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed" exitCode=0
Sep 08 00:33:43.664501 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.664441    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" event={"ID":"487de525-0f58-4d2d-964f-42b8355fab8d","Type":"ContainerDied","Data":"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"}
Sep 08 00:33:43.664501 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.664444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"
Sep 08 00:33:43.664501 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.664474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg" event={"ID":"487de525-0f58-4d2d-964f-42b8355fab8d","Type":"ContainerDied","Data":"6b4766d2316b72ee54977f61902186fcf8c3d71149648cf4340f25d19275686a"}
Sep 08 00:33:43.672123 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.672107    2580 scope.go:117] "RemoveContainer" containerID="36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"
Sep 08 00:33:43.672409 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:33:43.672392    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e\": container with ID starting with 36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e not found: ID does not exist" containerID="36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"
Sep 08 00:33:43.672495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.672419    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e"} err="failed to get container status \"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e\": rpc error: code = NotFound desc = could not find container \"36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e\": container with ID starting with 36cf414014726c9e77bac10741cb1f8d8435719fe8c8affaeabae087e3c8e66e not found: ID does not exist"
Sep 08 00:33:43.672495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.672440    2580 scope.go:117] "RemoveContainer" containerID="6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"
Sep 08 00:33:43.681647 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.681581    2580 scope.go:117] "RemoveContainer" containerID="6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"
Sep 08 00:33:43.681927 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:33:43.681906    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed\": container with ID starting with 6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed not found: ID does not exist" containerID="6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"
Sep 08 00:33:43.682006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.681935    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed"} err="failed to get container status \"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed\": rpc error: code = NotFound desc = could not find container \"6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed\": container with ID starting with 6a6ac7cd19c53f1b61005d0cbb53b4be1e7dcd22037c1771429eb02f0b73aeed not found: ID does not exist"
Sep 08 00:33:43.687398 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.687376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl2hg\" (UniqueName: \"kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg\") pod \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") "
Sep 08 00:33:43.687510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.687483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume\") pod \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\" (UID: \"99c1e1ac-9c30-47d0-9100-ee2f45ad2f92\") "
Sep 08 00:33:43.687839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.687813    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:43.687839 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.687834    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/487de525-0f58-4d2d-964f-42b8355fab8d-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:43.688032 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.687848    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6428m\" (UniqueName: \"kubernetes.io/projected/487de525-0f58-4d2d-964f-42b8355fab8d-kube-api-access-6428m\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:43.688225 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.688182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" (UID: "99c1e1ac-9c30-47d0-9100-ee2f45ad2f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:33:43.689953 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.689830    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg" (OuterVolumeSpecName: "kube-api-access-dl2hg") pod "99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" (UID: "99c1e1ac-9c30-47d0-9100-ee2f45ad2f92"). InnerVolumeSpecName "kube-api-access-dl2hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:33:43.734095 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.734057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:33:43.741308 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.741281    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d2f00-59d557f8b-tfcxg"]
Sep 08 00:33:43.788801 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.788728    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:43.788801 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:43.788753    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2hg\" (UniqueName: \"kubernetes.io/projected/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92-kube-api-access-dl2hg\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:33:44.001433 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:44.001401    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:33:44.013038 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:44.013008    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d2f00-75b7c44f94-mchdv"]
Sep 08 00:33:44.330940 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:44.330907    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487de525-0f58-4d2d-964f-42b8355fab8d" path="/var/lib/kubelet/pods/487de525-0f58-4d2d-964f-42b8355fab8d/volumes"
Sep 08 00:33:44.331314 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:33:44.331292    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" path="/var/lib/kubelet/pods/99c1e1ac-9c30-47d0-9100-ee2f45ad2f92/volumes"
Sep 08 00:34:04.315850 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.315812    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:34:04.318192 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.316021    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" podUID="74ade461-58f7-479c-812d-36a0d3381da3" containerName="maas-api" containerID="cri-o://965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c" gracePeriod=30
Sep 08 00:34:04.458820 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.458774    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" podUID="74ade461-58f7-479c-812d-36a0d3381da3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.39:8443/health\": dial tcp 10.134.0.39:8443: connect: connection refused"
Sep 08 00:34:04.593876 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.593851    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:34:04.681276 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca\") pod \"74ade461-58f7-479c-812d-36a0d3381da3\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") "
Sep 08 00:34:04.681442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs\") pod \"74ade461-58f7-479c-812d-36a0d3381da3\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") "
Sep 08 00:34:04.681539 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681505    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9t7s\" (UniqueName: \"kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s\") pod \"74ade461-58f7-479c-812d-36a0d3381da3\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") "
Sep 08 00:34:04.681617 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") pod \"74ade461-58f7-479c-812d-36a0d3381da3\" (UID: \"74ade461-58f7-479c-812d-36a0d3381da3\") "
Sep 08 00:34:04.681666 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74ade461-58f7-479c-812d-36a0d3381da3" (UID: "74ade461-58f7-479c-812d-36a0d3381da3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:34:04.681883 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.681865    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ade461-58f7-479c-812d-36a0d3381da3-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:04.683494 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.683471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "74ade461-58f7-479c-812d-36a0d3381da3" (UID: "74ade461-58f7-479c-812d-36a0d3381da3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:34:04.683723 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.683498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74ade461-58f7-479c-812d-36a0d3381da3" (UID: "74ade461-58f7-479c-812d-36a0d3381da3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:34:04.683723 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.683499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s" (OuterVolumeSpecName: "kube-api-access-n9t7s") pod "74ade461-58f7-479c-812d-36a0d3381da3" (UID: "74ade461-58f7-479c-812d-36a0d3381da3"). InnerVolumeSpecName "kube-api-access-n9t7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:34:04.760593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.760554    2580 generic.go:356] "Generic (PLEG): container finished" podID="74ade461-58f7-479c-812d-36a0d3381da3" containerID="965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c" exitCode=0
Sep 08 00:34:04.760593 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.760597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"
Sep 08 00:34:04.760785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.760625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" event={"ID":"74ade461-58f7-479c-812d-36a0d3381da3","Type":"ContainerDied","Data":"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"}
Sep 08 00:34:04.760785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.760705    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc" event={"ID":"74ade461-58f7-479c-812d-36a0d3381da3","Type":"ContainerDied","Data":"0227b32778ab10195cce1607f5cba5d5b7cbcc7179974be5c02d206e3526b0e5"}
Sep 08 00:34:04.760785 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.760732    2580 scope.go:117] "RemoveContainer" containerID="965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"
Sep 08 00:34:04.769765 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.769744    2580 scope.go:117] "RemoveContainer" containerID="965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"
Sep 08 00:34:04.770025 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:34:04.770001    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c\": container with ID starting with 965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c not found: ID does not exist" containerID="965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"
Sep 08 00:34:04.770096 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.770031    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c"} err="failed to get container status \"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c\": rpc error: code = NotFound desc = could not find container \"965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c\": container with ID starting with 965f8ca6574b0d365a97e10ebd2e240cbdd35059550fcd3b7b36a34d1ab5ea9c not found: ID does not exist"
Sep 08 00:34:04.783324 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.783297    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9t7s\" (UniqueName: \"kubernetes.io/projected/74ade461-58f7-479c-812d-36a0d3381da3-kube-api-access-n9t7s\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:04.783324 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.783323    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:04.783442 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.783331    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ade461-58f7-479c-812d-36a0d3381da3-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:04.797485 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.797454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:34:04.814426 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:04.814399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8978e7-6df96d7558-vpszc"]
Sep 08 00:34:06.331459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:06.331425    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ade461-58f7-479c-812d-36a0d3381da3" path="/var/lib/kubelet/pods/74ade461-58f7-479c-812d-36a0d3381da3/volumes"
Sep 08 00:34:25.603337 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.603260    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:34:25.603858 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.603477    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" podUID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" containerName="maas-api" containerID="cri-o://bc3c7d841454f0ef5de04ec8c17fb65e7dc32f64afae7da6669e09bc7d741650" gracePeriod=30
Sep 08 00:34:25.696588 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.696557    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:34:25.697003 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.696788    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" podUID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" containerName="payload-processing" containerID="cri-o://ec39b9dc44ac2c8cbe52c7037eed59e1a6f48ec50d71cc10ec877e8b42640239" gracePeriod=30
Sep 08 00:34:25.859863 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.859792    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" containerID="bc3c7d841454f0ef5de04ec8c17fb65e7dc32f64afae7da6669e09bc7d741650" exitCode=0
Sep 08 00:34:25.860096 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.859872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" event={"ID":"3a02dd05-affd-4ba1-9412-a5ab0108cb9f","Type":"ContainerDied","Data":"bc3c7d841454f0ef5de04ec8c17fb65e7dc32f64afae7da6669e09bc7d741650"}
Sep 08 00:34:25.861762 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.861736    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" containerID="ec39b9dc44ac2c8cbe52c7037eed59e1a6f48ec50d71cc10ec877e8b42640239" exitCode=0
Sep 08 00:34:25.861866 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.861777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" event={"ID":"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1","Type":"ContainerDied","Data":"ec39b9dc44ac2c8cbe52c7037eed59e1a6f48ec50d71cc10ec877e8b42640239"}
Sep 08 00:34:25.943641 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.943615    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:34:25.973126 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.973099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls\") pod \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") "
Sep 08 00:34:25.973268 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.973179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca\") pod \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") "
Sep 08 00:34:25.973268 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.973220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7p4h\" (UniqueName: \"kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h\") pod \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") "
Sep 08 00:34:25.973268 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.973253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs\") pod \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\" (UID: \"3a02dd05-affd-4ba1-9412-a5ab0108cb9f\") "
Sep 08 00:34:25.973683 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.973634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a02dd05-affd-4ba1-9412-a5ab0108cb9f" (UID: "3a02dd05-affd-4ba1-9412-a5ab0108cb9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:34:25.974367 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.974340    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:34:25.975709 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.975683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a02dd05-affd-4ba1-9412-a5ab0108cb9f" (UID: "3a02dd05-affd-4ba1-9412-a5ab0108cb9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:34:25.975798 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.975709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a02dd05-affd-4ba1-9412-a5ab0108cb9f" (UID: "3a02dd05-affd-4ba1-9412-a5ab0108cb9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:34:25.975798 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:25.975737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h" (OuterVolumeSpecName: "kube-api-access-t7p4h") pod "3a02dd05-affd-4ba1-9412-a5ab0108cb9f" (UID: "3a02dd05-affd-4ba1-9412-a5ab0108cb9f"). InnerVolumeSpecName "kube-api-access-t7p4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:34:26.074047 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume\") pod \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") "
Sep 08 00:34:26.074209 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4f5t\" (UniqueName: \"kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t\") pod \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\" (UID: \"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1\") "
Sep 08 00:34:26.074320 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074301    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.074402 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074324    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.074402 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074337    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7p4h\" (UniqueName: \"kubernetes.io/projected/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-kube-api-access-t7p4h\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.074402 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074346    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a02dd05-affd-4ba1-9412-a5ab0108cb9f-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.074559 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.074401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" (UID: "7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:34:26.076705 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.076674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t" (OuterVolumeSpecName: "kube-api-access-s4f5t") pod "7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" (UID: "7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1"). InnerVolumeSpecName "kube-api-access-s4f5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:34:26.175039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.175007    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-plugins-config-volume\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.175039 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.175035    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4f5t\" (UniqueName: \"kubernetes.io/projected/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1-kube-api-access-s4f5t\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:34:26.867813 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.867736    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"
Sep 08 00:34:26.867813 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.867754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj" event={"ID":"3a02dd05-affd-4ba1-9412-a5ab0108cb9f","Type":"ContainerDied","Data":"22c83f6e787329f1e46c56c03331fc4c62b690251aa489a1ab396396ba4cb90a"}
Sep 08 00:34:26.868276 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.867820    2580 scope.go:117] "RemoveContainer" containerID="bc3c7d841454f0ef5de04ec8c17fb65e7dc32f64afae7da6669e09bc7d741650"
Sep 08 00:34:26.869109 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.869095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql" event={"ID":"7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1","Type":"ContainerDied","Data":"bce9c5687e1450845720fec39f269eea831462c9ca84286a5cead1b30f07d519"}
Sep 08 00:34:26.869182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.869110    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"
Sep 08 00:34:26.876410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.876391    2580 scope.go:117] "RemoveContainer" containerID="ec39b9dc44ac2c8cbe52c7037eed59e1a6f48ec50d71cc10ec877e8b42640239"
Sep 08 00:34:26.890467 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.890444    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:34:26.896352 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.896329    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-b8c8cd-58459bd995-8xfdj"]
Sep 08 00:34:26.909481 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.909454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:34:26.916715 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:26.916692    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b8c8cd-7794689557-krwql"]
Sep 08 00:34:28.330137 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:28.330098    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" path="/var/lib/kubelet/pods/3a02dd05-affd-4ba1-9412-a5ab0108cb9f/volumes"
Sep 08 00:34:28.330505 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:34:28.330480    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" path="/var/lib/kubelet/pods/7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1/volumes"
Sep 08 00:39:34.468603 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"]
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468949    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" containerName="payload-processing"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468962    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" containerName="payload-processing"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468972    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487de525-0f58-4d2d-964f-42b8355fab8d" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468981    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="487de525-0f58-4d2d-964f-42b8355fab8d" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.468999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469008    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ade461-58f7-479c-812d-36a0d3381da3" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469013    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ade461-58f7-479c-812d-36a0d3381da3" containerName="maas-api"
Sep 08 00:39:34.469017 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469020    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457e56bb-79a9-412a-928d-0a695bcae88d" containerName="payload-pre-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="457e56bb-79a9-412a-928d-0a695bcae88d" containerName="payload-pre-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469036    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469042    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" containerName="payload-pre-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469059    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" containerName="payload-pre-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469067    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469072    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469119    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef2c13b-a5e5-4068-a1b1-1bf2fe9da1a1" containerName="payload-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469126    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c1e1ac-9c30-47d0-9100-ee2f45ad2f92" containerName="payload-pre-processing"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469137    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="847d4e5a-ad2f-4d9e-9996-37f67eb64fbb" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="487de525-0f58-4d2d-964f-42b8355fab8d" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469149    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcbd4f83-509d-48bd-af46-47a6a8da3e25" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469155    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a02dd05-affd-4ba1-9412-a5ab0108cb9f" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469163    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ade461-58f7-479c-812d-36a0d3381da3" containerName="maas-api"
Sep 08 00:39:34.469299 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.469167    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="457e56bb-79a9-412a-928d-0a695bcae88d" containerName="payload-pre-processing"
Sep 08 00:39:34.472006 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.471988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.482260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.482233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"]
Sep 08 00:39:34.643122 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.643089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlb6\" (UniqueName: \"kubernetes.io/projected/336c55e2-82fa-424f-a1b8-5c193d8c3699-kube-api-access-zqlb6\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.643360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.643166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336c55e2-82fa-424f-a1b8-5c193d8c3699-openshift-service-ca\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.643360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.643208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-metrics-certs\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.643360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.643238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-maas-api-tls\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.744770 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.744682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlb6\" (UniqueName: \"kubernetes.io/projected/336c55e2-82fa-424f-a1b8-5c193d8c3699-kube-api-access-zqlb6\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.744770 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.744760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336c55e2-82fa-424f-a1b8-5c193d8c3699-openshift-service-ca\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.744972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.744793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-metrics-certs\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.744972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.744828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-maas-api-tls\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.745587 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.745562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/336c55e2-82fa-424f-a1b8-5c193d8c3699-openshift-service-ca\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.747258 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.747236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-metrics-certs\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.747486 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.747464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/336c55e2-82fa-424f-a1b8-5c193d8c3699-maas-api-tls\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.752364 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.752345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlb6\" (UniqueName: \"kubernetes.io/projected/336c55e2-82fa-424f-a1b8-5c193d8c3699-kube-api-access-zqlb6\") pod \"maas-api-86b5f58bbf-fvl5g\" (UID: \"336c55e2-82fa-424f-a1b8-5c193d8c3699\") " pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:34.784249 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:34.784222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:35.065334 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:35.065306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"]
Sep 08 00:39:35.067128 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:39:35.067097    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336c55e2_82fa_424f_a1b8_5c193d8c3699.slice/crio-8659ba9733044eb192d8ea82a83b21fe6e493a775106aac15ccead35bfce9e14 WatchSource:0}: Error finding container 8659ba9733044eb192d8ea82a83b21fe6e493a775106aac15ccead35bfce9e14: Status 404 returned error can't find the container with id 8659ba9733044eb192d8ea82a83b21fe6e493a775106aac15ccead35bfce9e14
Sep 08 00:39:35.068649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:35.068626    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 00:39:35.198168 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:35.198136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g" event={"ID":"336c55e2-82fa-424f-a1b8-5c193d8c3699","Type":"ContainerStarted","Data":"8659ba9733044eb192d8ea82a83b21fe6e493a775106aac15ccead35bfce9e14"}
Sep 08 00:39:36.203847 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:36.203810    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g" event={"ID":"336c55e2-82fa-424f-a1b8-5c193d8c3699","Type":"ContainerStarted","Data":"9b17a4b532100c76dac8a7c023f732b6479939144ada42f92b89c873ae7b7855"}
Sep 08 00:39:36.204274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:36.203875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:36.229484 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:36.229423    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g" podStartSLOduration=1.799905566 podStartE2EDuration="2.22940495s" totalImagesPullingTime="429.499384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:36.228240916 +0000 UTC m=+2208.400505836" watchObservedRunningTime="2026-09-08 00:39:36.22940495 +0000 UTC m=+2208.401669833"
Sep 08 00:39:42.213994 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:42.213960    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86b5f58bbf-fvl5g"
Sep 08 00:39:49.977625 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:49.977583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:39:49.978150 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:49.977802    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" podUID="7b2dc967-8793-482c-820c-316efe11c091" containerName="maas-api" containerID="cri-o://8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b" gracePeriod=30
Sep 08 00:39:51.000805 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.000764    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" podUID="7b2dc967-8793-482c-820c-316efe11c091" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused"
Sep 08 00:39:51.222245 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.222219    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:39:51.273621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.270790    2580 generic.go:356] "Generic (PLEG): container finished" podID="7b2dc967-8793-482c-820c-316efe11c091" containerID="8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b" exitCode=0
Sep 08 00:39:51.273621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.270948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" event={"ID":"7b2dc967-8793-482c-820c-316efe11c091","Type":"ContainerDied","Data":"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"}
Sep 08 00:39:51.273621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.271030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g" event={"ID":"7b2dc967-8793-482c-820c-316efe11c091","Type":"ContainerDied","Data":"e3a0d98577ca2349f32eabdddac1fd7e95186a8e09f9acb486f4b8bf3ee06eaa"}
Sep 08 00:39:51.273621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.271072    2580 scope.go:117] "RemoveContainer" containerID="8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"
Sep 08 00:39:51.273621 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.271275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"
Sep 08 00:39:51.284453 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.284412    2580 scope.go:117] "RemoveContainer" containerID="8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"
Sep 08 00:39:51.284930 ip-10-0-128-150 kubenswrapper[2580]: E0908 00:39:51.284898    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b\": container with ID starting with 8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b not found: ID does not exist" containerID="8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"
Sep 08 00:39:51.285004 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.284948    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b"} err="failed to get container status \"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b\": rpc error: code = NotFound desc = could not find container \"8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b\": container with ID starting with 8610c9450ed613b786d273e1a7920720135940f747b02e785b64ce389652dd0b not found: ID does not exist"
Sep 08 00:39:51.290264 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.290238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") pod \"7b2dc967-8793-482c-820c-316efe11c091\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") "
Sep 08 00:39:51.290334 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.290299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8shtn\" (UniqueName: \"kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn\") pod \"7b2dc967-8793-482c-820c-316efe11c091\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") "
Sep 08 00:39:51.290383 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.290348    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") pod \"7b2dc967-8793-482c-820c-316efe11c091\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") "
Sep 08 00:39:51.290420 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.290379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca\") pod \"7b2dc967-8793-482c-820c-316efe11c091\" (UID: \"7b2dc967-8793-482c-820c-316efe11c091\") "
Sep 08 00:39:51.290786 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.290756    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b2dc967-8793-482c-820c-316efe11c091" (UID: "7b2dc967-8793-482c-820c-316efe11c091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 00:39:51.292367 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.292341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b2dc967-8793-482c-820c-316efe11c091" (UID: "7b2dc967-8793-482c-820c-316efe11c091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:39:51.292477 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.292397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b2dc967-8793-482c-820c-316efe11c091" (UID: "7b2dc967-8793-482c-820c-316efe11c091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 00:39:51.292477 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.292437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn" (OuterVolumeSpecName: "kube-api-access-8shtn") pod "7b2dc967-8793-482c-820c-316efe11c091" (UID: "7b2dc967-8793-482c-820c-316efe11c091"). InnerVolumeSpecName "kube-api-access-8shtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 00:39:51.391459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.391404    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-maas-api-tls\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:39:51.391459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.391451    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2dc967-8793-482c-820c-316efe11c091-openshift-service-ca\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:39:51.391459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.391463    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b2dc967-8793-482c-820c-316efe11c091-metrics-certs\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:39:51.391459 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.391472    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shtn\" (UniqueName: \"kubernetes.io/projected/7b2dc967-8793-482c-820c-316efe11c091-kube-api-access-8shtn\") on node \"ip-10-0-128-150.ec2.internal\" DevicePath \"\""
Sep 08 00:39:51.593868 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.593836    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:39:51.597214 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:51.597188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4c57c69-vnq6g"]
Sep 08 00:39:52.331398 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:39:52.331365    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2dc967-8793-482c-820c-316efe11c091" path="/var/lib/kubelet/pods/7b2dc967-8793-482c-820c-316efe11c091/volumes"
Sep 08 00:40:11.832097 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:11.832016    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gvgwg_9d404bdf-5cb8-40d7-90e6-cc9447cbc08d/server/0.log"
Sep 08 00:40:11.945972 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:11.945928    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6f5tf_c493d3b5-e6cf-45de-bc4f-822c202f5e64/manager/0.log"
Sep 08 00:40:12.290058 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:12.290030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cf67b67f-vpvdc_dac1dd37-4ffe-48d6-8d56-a12c480975d6/copy-manifests/0.log"
Sep 08 00:40:12.297485 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:12.297459    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cf67b67f-vpvdc_dac1dd37-4ffe-48d6-8d56-a12c480975d6/manager/0.log"
Sep 08 00:40:12.621126 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:12.621042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86b5f58bbf-fvl5g_336c55e2-82fa-424f-a1b8-5c193d8c3699/maas-api/0.log"
Sep 08 00:40:14.987510 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:14.987469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vj962_ff41e0b6-8375-4388-a167-f1ca67ff9d27/discovery/0.log"
Sep 08 00:40:16.022203 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:16.022164    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lj4hp_b5e6b966-aa3d-4286-b285-1dd75f78311c/main/0.log"
Sep 08 00:40:16.359343 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:16.359259    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6llh9_45612d10-3c98-4ba8-958b-7906fa8fb089/main/0.log"
Sep 08 00:40:16.467932 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:16.467897    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ffshpk_4c92334e-cbd6-4002-a49f-0fb4167bcb1f/main/0.log"
Sep 08 00:40:25.157936 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:25.157907    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xpvcr_b228f4d3-1a89-4fa8-87a0-78a71024d15e/global-pull-secret-syncer/0.log"
Sep 08 00:40:25.278582 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:25.278550    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qrfmj_0cbcd392-b02f-4a8b-b6b3-97713445f430/konnectivity-agent/0.log"
Sep 08 00:40:25.309423 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:25.309389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-150.ec2.internal_1b2683d5f56405395b041d9bac31f181/haproxy/0.log"
Sep 08 00:40:32.470612 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.470583    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpdvg_e69cd725-f8e7-42c0-8938-9c2a1a119e64/kube-state-metrics/0.log"
Sep 08 00:40:32.498283 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.498240    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpdvg_e69cd725-f8e7-42c0-8938-9c2a1a119e64/kube-rbac-proxy-main/0.log"
Sep 08 00:40:32.520677 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.520649    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-dpdvg_e69cd725-f8e7-42c0-8938-9c2a1a119e64/kube-rbac-proxy-self/0.log"
Sep 08 00:40:32.582048 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.582024    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-2th4n_b888ecc2-7649-4da3-9272-cdffeca466e3/monitoring-plugin/0.log"
Sep 08 00:40:32.696881 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.696854    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld88q_dfe8f8b5-d294-47ae-8592-40ec7f2b6989/node-exporter/0.log"
Sep 08 00:40:32.721566 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.721467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld88q_dfe8f8b5-d294-47ae-8592-40ec7f2b6989/kube-rbac-proxy/0.log"
Sep 08 00:40:32.743456 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.743429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld88q_dfe8f8b5-d294-47ae-8592-40ec7f2b6989/init-textfile/0.log"
Sep 08 00:40:32.865476 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.865441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kscxq_060c7c6e-871d-4a3f-b359-0e97a3c12aa0/kube-rbac-proxy-main/0.log"
Sep 08 00:40:32.887368 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.887346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kscxq_060c7c6e-871d-4a3f-b359-0e97a3c12aa0/kube-rbac-proxy-self/0.log"
Sep 08 00:40:32.911024 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.910998    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kscxq_060c7c6e-871d-4a3f-b359-0e97a3c12aa0/openshift-state-metrics/0.log"
Sep 08 00:40:32.949539 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.949501    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/prometheus/0.log"
Sep 08 00:40:32.971360 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.971338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/config-reloader/0.log"
Sep 08 00:40:32.996106 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:32.996042    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/thanos-sidecar/0.log"
Sep 08 00:40:33.019774 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:33.019750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/kube-rbac-proxy-web/0.log"
Sep 08 00:40:33.044728 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:33.044698    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/kube-rbac-proxy/0.log"
Sep 08 00:40:33.069599 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:33.069577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/kube-rbac-proxy-thanos/0.log"
Sep 08 00:40:33.097505 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:33.097483    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e64ffbf-542b-4ba3-863c-34f04da30d57/init-config-reloader/0.log"
Sep 08 00:40:34.809867 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.809830    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"]
Sep 08 00:40:34.810381 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.810364    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2dc967-8793-482c-820c-316efe11c091" containerName="maas-api"
Sep 08 00:40:34.810434 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.810386    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2dc967-8793-482c-820c-316efe11c091" containerName="maas-api"
Sep 08 00:40:34.810486 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.810475    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2dc967-8793-482c-820c-316efe11c091" containerName="maas-api"
Sep 08 00:40:34.813909 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.813886    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:34.816390 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.816372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hsrk2\"/\"default-dockercfg-pzcbl\""
Sep 08 00:40:34.816490 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.816444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsrk2\"/\"openshift-service-ca.crt\""
Sep 08 00:40:34.817410 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.817395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hsrk2\"/\"kube-root-ca.crt\""
Sep 08 00:40:34.821494 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.821468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"]
Sep 08 00:40:34.966514 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.966474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfx2\" (UniqueName: \"kubernetes.io/projected/a2560b87-6bbf-4e29-abce-98cd2f257235-kube-api-access-vkfx2\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:34.966726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.966600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-podres\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:34.966726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.966630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-proc\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:34.966726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.966648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-lib-modules\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:34.966726 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:34.966664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-sys\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.067825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-proc\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.067825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-lib-modules\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.067825 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-sys\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfx2\" (UniqueName: \"kubernetes.io/projected/a2560b87-6bbf-4e29-abce-98cd2f257235-kube-api-access-vkfx2\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-proc\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-sys\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.067977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-lib-modules\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068063 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.068003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-podres\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.068274 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.068109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2560b87-6bbf-4e29-abce-98cd2f257235-podres\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.084520 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.084497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfx2\" (UniqueName: \"kubernetes.io/projected/a2560b87-6bbf-4e29-abce-98cd2f257235-kube-api-access-vkfx2\") pod \"perf-node-gather-daemonset-zst52\" (UID: \"a2560b87-6bbf-4e29-abce-98cd2f257235\") " pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.124667 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.124626    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.394182 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.394157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"]
Sep 08 00:40:35.395988 ip-10-0-128-150 kubenswrapper[2580]: W0908 00:40:35.395960    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2560b87_6bbf_4e29_abce_98cd2f257235.slice/crio-24fd57a33eae51d6f9dd33733f4bebc93dfcf7201b4070d4d8d57cb726e72018 WatchSource:0}: Error finding container 24fd57a33eae51d6f9dd33733f4bebc93dfcf7201b4070d4d8d57cb726e72018: Status 404 returned error can't find the container with id 24fd57a33eae51d6f9dd33733f4bebc93dfcf7201b4070d4d8d57cb726e72018
Sep 08 00:40:35.457556 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.457510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52" event={"ID":"a2560b87-6bbf-4e29-abce-98cd2f257235","Type":"ContainerStarted","Data":"04179e6c48ecc9751a890cbe52cb1edffff7a9045446a4c7815b77ed88ea7021"}
Sep 08 00:40:35.457649 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.457578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52" event={"ID":"a2560b87-6bbf-4e29-abce-98cd2f257235","Type":"ContainerStarted","Data":"24fd57a33eae51d6f9dd33733f4bebc93dfcf7201b4070d4d8d57cb726e72018"}
Sep 08 00:40:35.457703 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.457656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:35.473341 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:35.473290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52" podStartSLOduration=1.473275626 podStartE2EDuration="1.473275626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:40:35.471753158 +0000 UTC m=+2267.644018039" watchObservedRunningTime="2026-09-08 00:40:35.473275626 +0000 UTC m=+2267.645540510"
Sep 08 00:40:37.717208 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:37.717177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nshh7_57937680-0364-41f0-b9ae-39a9d7c0a1cd/dns/0.log"
Sep 08 00:40:37.737915 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:37.737889    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nshh7_57937680-0364-41f0-b9ae-39a9d7c0a1cd/kube-rbac-proxy/0.log"
Sep 08 00:40:37.818044 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:37.818013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5f47g_c0b23f53-7ab5-4440-bb3f-9cb0b3b7e997/dns-node-resolver/0.log"
Sep 08 00:40:38.476013 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:38.475976    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-665fcd494-75bck_fdd3e2b9-1d73-4933-930a-c56c09b2e5e7/registry/0.log"
Sep 08 00:40:38.527901 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:38.527871    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ppdfm_7f1e5563-87ef-49d6-b328-2b208ea3da80/node-ca/0.log"
Sep 08 00:40:40.042964 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:40.042925    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vj962_ff41e0b6-8375-4388-a167-f1ca67ff9d27/discovery/0.log"
Sep 08 00:40:41.154783 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:41.154750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tdw48_c0fd4558-3d7c-44d3-b67d-19c541d6b714/serve-healthcheck-canary/0.log"
Sep 08 00:40:41.471622 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:41.471548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hsrk2/perf-node-gather-daemonset-zst52"
Sep 08 00:40:41.729242 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:41.729158    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4kjq_37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de/kube-rbac-proxy/0.log"
Sep 08 00:40:41.750260 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:41.750232    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4kjq_37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de/exporter/0.log"
Sep 08 00:40:41.771219 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:41.771183    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v4kjq_37a01fa5-4cf7-4d65-ab7b-a1b1bc5726de/extractor/0.log"
Sep 08 00:40:44.395781 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:44.395736    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-gvgwg_9d404bdf-5cb8-40d7-90e6-cc9447cbc08d/server/0.log"
Sep 08 00:40:44.553714 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:44.553673    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6f5tf_c493d3b5-e6cf-45de-bc4f-822c202f5e64/manager/0.log"
Sep 08 00:40:44.884923 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:44.884886    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cf67b67f-vpvdc_dac1dd37-4ffe-48d6-8d56-a12c480975d6/manager/0.log"
Sep 08 00:40:44.915155 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:44.915122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57cf67b67f-vpvdc_dac1dd37-4ffe-48d6-8d56-a12c480975d6/copy-manifests/0.log"
Sep 08 00:40:45.561144 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:45.561116    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vj962_ff41e0b6-8375-4388-a167-f1ca67ff9d27/discovery/0.log"
Sep 08 00:40:46.363474 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:46.363446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c54b7b87b-bbvx5_e78f5779-087b-4a14-9505-f4f2e616657f/manager/0.log"
Sep 08 00:40:47.892158 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:47.892070    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-vj962_ff41e0b6-8375-4388-a167-f1ca67ff9d27/discovery/0.log"
Sep 08 00:40:51.758503 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:51.758458    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fn8qz_997ae1d7-00b3-4316-83a0-f116a4e93eae/migrator/0.log"
Sep 08 00:40:51.780369 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:51.780336    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fn8qz_997ae1d7-00b3-4316-83a0-f116a4e93eae/graceful-termination/0.log"
Sep 08 00:40:53.404224 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:53.404199    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fc9f2_4ed3c48f-fe87-489a-b011-76cb7b05fe30/kube-multus-additional-cni-plugins/0.log"
Sep 08 00:40:53.426495 ip-10-0-128-150 kubenswrapper[2580]: I0908 00:40:53.426467    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fc9f2_4ed3c48f-fe87-489a-b011-76cb7b05fe30/egress-router-binary-copy/0.log"