Sep 09 16:11:45.463153 ip-10-0-133-104 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 16:11:45.463163 ip-10-0-133-104 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 16:11:45.463170 ip-10-0-133-104 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 16:11:45.463384 ip-10-0-133-104 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 16:11:54.787434 ip-10-0-133-104 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 09 16:11:54.787446 ip-10-0-133-104 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 168c2fd60b30485c9fe16b9acb172374 --
Sep 09 16:13:38.254695 ip-10-0-133-104 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 16:13:38.697214 ip-10-0-133-104 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:38.697214 ip-10-0-133-104 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 16:13:38.697214 ip-10-0-133-104 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:38.697214 ip-10-0-133-104 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 16:13:38.697214 ip-10-0-133-104 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 16:13:38.700065 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.699971    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 16:13:38.704851 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704836    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.704851 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704852    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704856    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704860    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704863    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704866    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704869    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704872    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704875    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704877    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704880    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704883    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704900    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704903    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704906    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704908    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704911    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704913    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704916    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704919    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704921    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.704916 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704924    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704927    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704930    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704934    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704938    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704941    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704944    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704947    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704949    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704952    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704954    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704957    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704959    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704962    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704966    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704970    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704974    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704977    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704980    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.705382 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704984    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704987    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704990    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704993    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704996    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.704998    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705001    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705003    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705006    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705008    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705011    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705013    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705016    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705018    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705021    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705025    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705028    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705030    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705033    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.705847 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705036    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705038    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705040    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705043    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705045    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705048    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705050    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705053    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705056    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705058    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705060    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705063    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705067    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705069    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705072    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705074    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705076    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705079    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705081    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705084    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.706325 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705086    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705089    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705092    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705094    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705096    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705099    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705101    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705104    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705107    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705110    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705113    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705115    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705118    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705120    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705123    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705126    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705128    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705131    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705133    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705136    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.706821 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705852    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705858    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705862    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705865    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705868    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705871    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705874    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705876    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705879    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705881    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705884    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705886    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705889    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705891    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705893    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705896    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705899    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705901    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705904    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705906    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.707283 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705909    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705913    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705915    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705918    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705921    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705923    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705925    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705928    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705930    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705933    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705935    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705938    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705941    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705943    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705945    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705948    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705952    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705956    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705959    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705962    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.707775 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705965    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705967    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705970    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705973    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705976    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705980    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705983    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705985    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705988    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705991    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705993    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.705996    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706000    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706003    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706006    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706009    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706012    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706015    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706018    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.708243 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706020    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706023    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706025    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706028    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706032    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706034    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706037    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706039    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706043    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706045    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706048    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706050    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706053    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706056    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706058    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706061    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706063    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706066    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706068    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.708719 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706071    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706073    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706076    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706078    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706080    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706083    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706088    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706091    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706094    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706096    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706099    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706101    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706104    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706106    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706109    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706111    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706114    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706116    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706119    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706121    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.709163 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706124    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706499    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706504    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706506    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706509    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706513    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706516    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706518    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706521    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706524    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706526    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706529    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706531    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706534    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706536    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706539    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706542    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706545    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706550    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706552    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.709651 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706555    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706557    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706560    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706562    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706565    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706567    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706570    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706573    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706575    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706577    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706580    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706582    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706585    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706588    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706590    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706593    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706595    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706598    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706601    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.710116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706604    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706607    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706609    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706612    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706615    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706619    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706621    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706624    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706626    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706629    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706631    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706635    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706637    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706640    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706643    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706645    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706648    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706652    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706654    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706657    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.710589 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706659    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706662    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706670    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706673    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706676    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706679    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706681    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706684    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706686    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706690    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706694    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706697    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706701    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706704    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706706    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706709    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706711    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706716    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706719    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.711116 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706722    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706725    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706728    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706730    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706735    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706738    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706740    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706743    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706745    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706748    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706752    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706755    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706757    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706760    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706762    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706765    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706767    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706770    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706772    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706775    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.711574 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706777    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.706780    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706846    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706890    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706896    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706901    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706906    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706909    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706913    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706917    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706920    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706923    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706927    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706930    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706933    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706935    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706940    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706943    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706946    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706949    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706954    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706957    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706960    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706963    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 16:13:38.712042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706967    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706970    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706973    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706976    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706979    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706982    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706985    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706988    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706991    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706993    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.706998    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707000    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707003    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707006    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707009    2559 flags.go:64] FLAG: --enable-server="true"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707012    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707016    2559 flags.go:64] FLAG: --event-burst="100"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707019    2559 flags.go:64] FLAG: --event-qps="50"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707022    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707025    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707028    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707031    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707034    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707037    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707041    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 09 16:13:38.712628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707044    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707047    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707049    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707052    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707055    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707058    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707061    2559 flags.go:64] FLAG: --feature-gates=""
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707065    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707068    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707071    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707074    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707077    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707080    2559 flags.go:64] FLAG: --help="false"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707083    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707085    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707088    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707091    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707094    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707098    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707100    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707103    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707106    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707109    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 16:13:38.713206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707113    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707116    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707118    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707121    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707124    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707127    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707129    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707132    2559 flags.go:64] FLAG: --lock-file=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707137    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707139    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707142    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707148    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707150    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707153    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707156    2559 flags.go:64] FLAG: --logging-format="text"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707159    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707162    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707165    2559 flags.go:64] FLAG: --manifest-url=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707168    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707172    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707175    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707179    2559 flags.go:64] FLAG: --max-pods="110"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707182    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707185    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707187    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 16:13:38.713794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707190    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707193    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707196    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707198    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707205    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707208    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707211    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707214    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707217    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707223    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707226    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707229    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707232    2559 flags.go:64] FLAG: --port="10250"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707235    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707238    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-061a1c26d1dab30f3"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707241    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707244    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707247    2559 flags.go:64] FLAG: --register-node="true"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707250    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707253    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707256    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707259    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707262    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707265    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707268    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 16:13:38.714423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707271    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707274    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707277    2559 flags.go:64] FLAG: --runonce="false"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707279    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707282    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707286    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707289    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707292    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707294    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707298    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707300    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707303    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707306    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707309    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707312    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707315    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707318    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707321    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707326    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707328    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707331    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707336    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707338    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707342    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707344    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 16:13:38.715102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707347    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707350    2559 flags.go:64] FLAG: --v="2"
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707354    2559 flags.go:64] FLAG: --version="false"
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707358    2559 flags.go:64] FLAG: --vmodule=""
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707362    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.707365    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707496    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707500    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707503    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707505    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707508    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707511    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707513    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707516    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707518    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707521    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707523    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707526    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707528    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707531    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707533    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.715715 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707536    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707538    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707541    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707544    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707546    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707549    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707551    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707553    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707556    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707559    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707562    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707564    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707567    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707569    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707573    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707576    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707578    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707581    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707583    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707586    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.716214 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707588    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707590    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707593    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707595    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707599    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707602    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707606    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707609    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707611    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707614    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707617    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707619    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707622    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707624    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707627    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707630    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707633    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707635    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707638    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.716745 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707640    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707643    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707645    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707648    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707650    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707652    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707655    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707659    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707662    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707664    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707667    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707670    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707672    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707675    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707679    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707682    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707685    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707688    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707690    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.717179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707693    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707696    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707698    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707701    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707703    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707705    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707708    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707710    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707713    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707715    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707718    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707720    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707723    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707726    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707728    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707731    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707733    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707735    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707745    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.717627 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707749    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707754    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707757    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707759    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707762    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707764    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.707767    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.708539    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.715832    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.715844    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.716973    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.716980    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717034    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717039    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717042    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.718117 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717045    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717048    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717051    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717054    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717057    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717060    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717063    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717065    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717067    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717071    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717074    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717077    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717080    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717083    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717086    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717088    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717091    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717093    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717097    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717100    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.718493 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717103    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717105    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717108    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717110    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717113    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717115    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717118    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717120    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717123    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717125    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717128    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717130    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717133    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717135    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717139    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717142    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717146    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717149    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717152    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.718967 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717155    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717157    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717160    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717163    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717165    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717168    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717171    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717173    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717176    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717178    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717181    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717184    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717186    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717189    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717192    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717194    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717198    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717202    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717205    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717207    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.719419 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717210    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717212    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717214    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717217    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717219    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717222    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717224    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717226    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717229    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717231    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717234    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717236    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717239    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717241    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717244    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717247    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717250    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717253    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717255    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.719923 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717257    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717260    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717263    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717265    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717268    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717270    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717273    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717275    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717277    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717280    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717282    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717285    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717287    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717289    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717292    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717294    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717297    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.720363 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717299    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.717304    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717410    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717414    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717417    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717419    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717422    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717425    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717442    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717448    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717452    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717455    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717458    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717461    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717464    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 16:13:38.720845 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717466    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717469    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717472    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717474    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717477    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717479    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717482    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717484    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717487    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717490    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717492    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717495    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717497    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717499    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717502    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717504    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717507    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717509    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717513    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717516    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 16:13:38.721197 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717519    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717522    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717525    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717528    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717530    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717533    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717536    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717538    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717540    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717543    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717545    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717549    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717551    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717554    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717557    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717559    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717562    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717564    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717567    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717569    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 16:13:38.721687 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717571    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717574    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717577    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717579    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717581    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717584    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717587    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717590    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717593    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717595    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717598    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717600    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717603    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717605    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717608    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717610    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717612    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717615    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717617    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 16:13:38.722144 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717620    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717622    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717625    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717627    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717630    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717633    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717635    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717638    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717640    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717643    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717645    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717648    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717650    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717653    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717655    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717657    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717660    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717662    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717665    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717667    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 16:13:38.722590 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717669    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717672    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717674    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717676    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717679    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717681    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:38.717683    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.717688    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.717774    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.720034    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.720984    2559 server.go:1019] "Starting client certificate rotation"
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.721538    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:13:38.723092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.721571    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 16:13:38.724145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.724132    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:13:38.731987 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.731968    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 16:13:38.742298 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.742284    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 09 16:13:38.745539 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.745523    2559 log.go:25] "Validated CRI v1 image API"
Sep 09 16:13:38.746314 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.746298    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 16:13:38.748951 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.748933    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2c1706e-d2e4-4e12-97a8-4a04e90e44b4:/dev/nvme0n1p3 d3ad9742-6ed3-4c36-ac97-b934bde76a8b:/dev/nvme0n1p4]
Sep 09 16:13:38.749032 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.748951    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 16:13:38.754041 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.753931    2559 manager.go:216] Machine: {Timestamp:2026-09-09 16:13:38.752565489 +0000 UTC m=+0.370674248 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3208307 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec264fcb27522aac01cf4edb7b165d77 SystemUUID:ec264fcb-2752-2aac-01cf-4edb7b165d77 BootID:168c2fd6-0b30-485c-9fe1-6b9acb172374 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:78:f9:65:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:78:f9:65:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:67:a2:8e:3d:a8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 16:13:38.754041 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.754034    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 16:13:38.754183 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.754166    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 16:13:38.755006 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.754991    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:13:38.757473 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.757450    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 16:13:38.757597 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.757475    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 16:13:38.758333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.758324    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 16:13:38.758363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.758336    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 16:13:38.758363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.758343    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:13:38.758363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.758357    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 16:13:38.760130 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.760120    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 16:13:38.763403 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763392    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:13:38.763523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763514    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 16:13:38.763558 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763542    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 16:13:38.763558 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763553    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 16:13:38.763611 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763570    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 16:13:38.763611 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763582    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 09 16:13:38.763611 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763592    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 16:13:38.763909 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763898    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:13:38.763958 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.763915    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 16:13:38.766187 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.766170    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 16:13:38.769115 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769101    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 16:13:38.769172 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769130    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 16:13:38.769280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769271    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 16:13:38.769309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769285    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 16:13:38.769309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769292    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 16:13:38.769309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769297    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 16:13:38.769309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769305    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 16:13:38.769309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769310    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 16:13:38.769914 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769904    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 16:13:38.769914 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769915    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 16:13:38.769978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769921    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 16:13:38.769978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769927    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 16:13:38.769978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769941    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 16:13:38.769978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.769948    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 16:13:38.770914 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.770882    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 16:13:38.771079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.771067    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 16:13:38.774979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.774956    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 16:13:38.775548 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.775535    2559 server.go:1268] "Started kubelet"
Sep 09 16:13:38.775652 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.775606    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 16:13:38.775713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.775671    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 16:13:38.775713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.775670    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 16:13:38.776072 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.776047    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 16:13:38.776131 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.776047    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 16:13:38.776185 ip-10-0-133-104 systemd[1]: Started Kubernetes Kubelet.
Sep 09 16:13:38.776926 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.776911    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 16:13:38.777400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.777380    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-szg4z"
Sep 09 16:13:38.777888 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.777816    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 16:13:38.780424 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.780403    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 16:13:38.780796 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.779748    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-104.ec2.internal.18d3b31e14cba972  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-104.ec2.internal,UID:ip-10-0-133-104.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-104.ec2.internal,},FirstTimestamp:2026-09-09 16:13:38.77550117 +0000 UTC m=+0.393609929,LastTimestamp:2026-09-09 16:13:38.77550117 +0000 UTC m=+0.393609929,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-104.ec2.internal,}"
Sep 09 16:13:38.784046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.784030    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-szg4z"
Sep 09 16:13:38.784647 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.784628    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 16:13:38.785270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.785256    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 16:13:38.785379 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.785351    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:38.785997 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.785974    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 16:13:38.785997 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.785993    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 16:13:38.786084 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.785973    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 16:13:38.786123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786081    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 16:13:38.786123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786092    2559 factory.go:55] Registering systemd factory
Sep 09 16:13:38.786123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786095    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 16:13:38.786123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786099    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 09 16:13:38.786123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786102    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 16:13:38.786308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786296    2559 factory.go:153] Registering CRI-O factory
Sep 09 16:13:38.786337 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786311    2559 factory.go:223] Registration of the crio container factory successfully
Sep 09 16:13:38.786337 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786326    2559 factory.go:103] Registering Raw factory
Sep 09 16:13:38.786398 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786340    2559 manager.go:1196] Started watching for new ooms in manager
Sep 09 16:13:38.786777 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.786764    2559 manager.go:349] Starting recovery of all containers
Sep 09 16:13:38.794780 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.794749    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:38.798650 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.798626    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-104.ec2.internal\" not found" node="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.799649 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.799633    2559 manager.go:354] Recovery completed
Sep 09 16:13:38.803866 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.803851    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.805794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.805779    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.805865 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.805810    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.805865 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.805824    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.806372 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.806358    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 16:13:38.806372 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.806365    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.806372 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.806371    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 16:13:38.806507 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.806386    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 16:13:38.807327 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807316    2559 policy_none.go:49] "None policy: Start"
Sep 09 16:13:38.807365 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807330    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 16:13:38.807365 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807339    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 16:13:38.807937 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807914    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.807937 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807938    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.808027 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.807947    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.808136 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.808125    2559 policy_none.go:47] "Start"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.848965    2559 manager.go:337] "Starting Device Plugin manager"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.848999    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849013    2559 server.go:87] "Starting device plugin registration server"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849249    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849260    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849376    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849478    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.849485    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.849942    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 16:13:38.853310 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.849980    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:38.855019 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.854994    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 16:13:38.856190 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.856171    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 16:13:38.856291 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.856227    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 16:13:38.856291 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.856250    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 16:13:38.856387 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.856292    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 16:13:38.858387 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.858370    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:38.950463 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.950356    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.951321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.951302    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.951404 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.951335    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.951404 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.951348    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.951404 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.951372    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.957258 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.957242    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.957311 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.957264    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-104.ec2.internal\": node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:38.957358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.957341    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"]
Sep 09 16:13:38.957414 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.957406    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.958884 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.958872    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.958935 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.958901    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.958935 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.958911    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.960011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960000    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.960122 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.960161 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960140    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.960720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960697    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.960720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960722    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.960858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960744    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.960858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960757    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.960858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960726    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.960858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.960830    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.961976 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.961959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.962070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.961988    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 16:13:38.962648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.962634    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 16:13:38.962745 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.962657    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 16:13:38.962745 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.962676    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeHasSufficientPID"
Sep 09 16:13:38.970240 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.970221    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:38.987880 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.987860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a982754da1ab91809fe032c16435bd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-104.ec2.internal\" (UID: \"92a982754da1ab91809fe032c16435bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.987948 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.987882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.987948 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:38.987899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.991054 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.991037    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-104.ec2.internal\" not found" node="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:38.995259 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:38.995245    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-104.ec2.internal\" not found" node="ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.071154 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.071131    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.089013 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.088987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.089099 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.089016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.089099 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.089033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a982754da1ab91809fe032c16435bd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-104.ec2.internal\" (UID: \"92a982754da1ab91809fe032c16435bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.089180 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.089105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/92a982754da1ab91809fe032c16435bd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-104.ec2.internal\" (UID: \"92a982754da1ab91809fe032c16435bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.089180 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.089108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.089180 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.089109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db996e25865c7c90759e7b27b65ff0d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal\" (UID: \"db996e25865c7c90759e7b27b65ff0d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.172264 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.172224    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.273009 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.272947    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.294337 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.294311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.299622 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.299606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.373888 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.373748    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.474293 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.474261    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.574733 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:39.574689    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-104.ec2.internal\" not found"
Sep 09 16:13:39.625382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.625363    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:39.686388 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.686370    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.697066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.697048    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:13:39.698994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.698981    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal"
Sep 09 16:13:39.706572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.706559    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 16:13:39.720716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.720699    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 16:13:39.720811 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.720795    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:39.720867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.720807    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:39.720867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.720812    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 16:13:39.764001 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:39.763977    2559 apiserver.go:52] "Watching apiserver"
Sep 09 16:13:41.450436 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.450419    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 16:13:41.450711 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.450693    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bxkn8","kube-system/global-pull-secret-syncer-5btpt","openshift-cluster-node-tuning-operator/tuned-8jff9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal","openshift-multus/multus-additional-cni-plugins-mqmsm","openshift-multus/multus-cnfqg","openshift-network-diagnostics/network-check-target-f5tf2","kube-system/konnectivity-agent-hqpjn","kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct","openshift-dns/node-resolver-cm49p","openshift-image-registry/node-ca-9w9h5","openshift-multus/network-metrics-daemon-mgf65","openshift-network-operator/iptables-alerter-mjkz5"]
Sep 09 16:13:41.453247 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.453229    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 16:13:41.453335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.453249    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.454631 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.454588    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 16:08:40 +0000 UTC" deadline="2028-06-24 20:39:09.734521404 +0000 UTC"
Sep 09 16:13:41.454631 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.454630    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15700h25m28.279894733s"
Sep 09 16:13:41.455305 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.455278    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.455399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.455358    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.455600 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.455580    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 16:13:41.455681 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.455588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j29t9\""
Sep 09 16:13:41.455760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.455742    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.455856 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.455822    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:41.458119 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.458099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.460007 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.459991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.460098 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.460049    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.460156 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.460121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b62br\""
Sep 09 16:13:41.460156 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.460128    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.461953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.461935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-registration-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462043 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.461982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462043 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.462043 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssxf\" (UniqueName: \"kubernetes.io/projected/c6337926-86fe-44e6-82e4-2e7637dc1b70-kube-api-access-bssxf\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.462189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-socket-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-device-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-sys-fs\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6h6m\" (UniqueName: \"kubernetes.io/projected/dd64f617-1030-43b9-bf58-a4e01eaed827-kube-api-access-q6h6m\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.462385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.462385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462285    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 16:13:41.462475 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462386    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.462634 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462620    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.462791 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.462776    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hxb6n\""
Sep 09 16:13:41.464556 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464542    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.464639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2v25j\""
Sep 09 16:13:41.464703 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.464759 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464709    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 16:13:41.464848 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464807    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 16:13:41.464950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.464884    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 16:13:41.465090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.465075    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 16:13:41.465157 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.465147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.466987 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.466971    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4kfhd\""
Sep 09 16:13:41.467031 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.467009    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 16:13:41.467092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.467079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.469307 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.469168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 16:13:41.469407 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.469361    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7wbz9\""
Sep 09 16:13:41.469467 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.469404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 16:13:41.469467 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.469433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:41.469555 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.469486    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:41.471537 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.471520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.471615 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.471578    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:41.473804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.473789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.477342 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477285    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.477449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477388    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 16:13:41.477449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477395    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.477559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.477559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477499    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvgrk\""
Sep 09 16:13:41.477559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477391    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 16:13:41.477677 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477611    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 16:13:41.477905 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.477805    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 16:13:41.479535 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.479520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.479760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.479741    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9fcc\""
Sep 09 16:13:41.480085 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.480072    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.480494 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.480480    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.481487 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.481473    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 16:13:41.481571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.481490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 16:13:41.481802 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.481786    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v855q\""
Sep 09 16:13:41.482002 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.481990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 16:13:41.483799 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.483779    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-krf7s"
Sep 09 16:13:41.494401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.494354    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-krf7s"
Sep 09 16:13:41.552901 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.552880    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a982754da1ab91809fe032c16435bd.slice/crio-b1bac193c8025016c3716f6f0b45c77d935120da63c5d2bce98f24668c289f6a WatchSource:0}: Error finding container b1bac193c8025016c3716f6f0b45c77d935120da63c5d2bce98f24668c289f6a: Status 404 returned error can't find the container with id b1bac193c8025016c3716f6f0b45c77d935120da63c5d2bce98f24668c289f6a
Sep 09 16:13:41.555433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.555413    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 16:13:41.557059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.557027    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:13:41.562338 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562322    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-socket-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.562420 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-host\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.562420 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562379    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4s9\" (UniqueName: \"kubernetes.io/projected/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-kube-api-access-pf4s9\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.562420 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.562563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.562563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-conf-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.562563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-socket-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.562592    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562595    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-kubernetes\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-sys\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6tq\" (UniqueName: \"kubernetes.io/projected/0ef9acfc-b406-4597-b256-061812bcd27d-kube-api-access-2x6tq\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.562659    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:42.062638339 +0000 UTC m=+2.012219180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:41.562701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjtrr\" (UniqueName: \"kubernetes.io/projected/9c0a1341-9866-4678-acf3-753b34a41424-kube-api-access-bjtrr\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-kubelet\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562786    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-env-overrides\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4n6\" (UniqueName: \"kubernetes.io/projected/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-kube-api-access-sj4n6\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2821554a-7a86-463f-a3bf-78c92fd9cba0-serviceca\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.562996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-conf\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.563300 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.562914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-registration-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.563427 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563379    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-registration-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.563489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-systemd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563536 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-var-lib-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-config\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563594    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-tmp\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.563672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-device-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.563672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-socket-dir-parent\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.563795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-systemd-units\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-bin\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.563795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysconfig\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.563795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7513a562-0c6a-4272-a1b9-030f19d3600d-host-slash\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.563795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-kubelet-config\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2821554a-7a86-463f-a3bf-78c92fd9cba0-host\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6h6m\" (UniqueName: \"kubernetes.io/projected/dd64f617-1030-43b9-bf58-a4e01eaed827-kube-api-access-q6h6m\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-slash\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.563975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-etc-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.564021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssxf\" (UniqueName: \"kubernetes.io/projected/c6337926-86fe-44e6-82e4-2e7637dc1b70-kube-api-access-bssxf\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-system-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-multus\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs4b\" (UniqueName: \"kubernetes.io/projected/2821554a-7a86-463f-a3bf-78c92fd9cba0-kube-api-access-jjs4b\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-device-dir\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-lib-modules\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.564296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-k8s-cni-cncf-io\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-hostroot\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-etc-kubernetes\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-netns\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-log-socket\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d07d2f8-cd59-4b61-8b13-27675f985d5b-tmp-dir\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.564579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-netns\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.564932 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.565074 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.564565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-ovn\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.565145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovn-node-metrics-cert\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.565145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-systemd\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.565232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-cnibin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.565232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-multus-daemon-config\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.565232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-multus-certs\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.565232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-dbus\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjfc\" (UniqueName: \"kubernetes.io/projected/7513a562-0c6a-4272-a1b9-030f19d3600d-kube-api-access-dhjfc\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-system-cni-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565308    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cnibin\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d07d2f8-cd59-4b61-8b13-27675f985d5b-hosts-file\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-cni-binary-copy\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.565399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.565661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565413    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-modprobe-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.565661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ln6\" (UniqueName: \"kubernetes.io/projected/6d07d2f8-cd59-4b61-8b13-27675f985d5b-kube-api-access-25ln6\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.565661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:41.565661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-kubelet\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.565661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-script-lib\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-run\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-etc-tuned\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-sys-fs\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.565869 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-node-log\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-agent-certs\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd64f617-1030-43b9-bf58-a4e01eaed827-sys-fs\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.565925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-bin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.566011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-netd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.566043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-var-lib-kubelet\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.566073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7513a562-0c6a-4272-a1b9-030f19d3600d-iptables-alerter-script\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.566131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.566099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-konnectivity-ca\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.570755 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.570739    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 16:13:41.573859 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.573837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6h6m\" (UniqueName: \"kubernetes.io/projected/dd64f617-1030-43b9-bf58-a4e01eaed827-kube-api-access-q6h6m\") pod \"aws-ebs-csi-driver-node-rlsct\" (UID: \"dd64f617-1030-43b9-bf58-a4e01eaed827\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.573927 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.573904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssxf\" (UniqueName: \"kubernetes.io/projected/c6337926-86fe-44e6-82e4-2e7637dc1b70-kube-api-access-bssxf\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:41.666385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-kubelet\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.666385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-script-lib\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.666571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-run\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.666571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-etc-tuned\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.666571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-kubelet\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.666571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-run\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.666716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.666716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.666716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-node-log\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.666716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-agent-certs\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-bin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-netd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.666763    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-node-log\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-var-lib-kubelet\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-bin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-var-lib-kubelet\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-netd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.666815    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:42.166796217 +0000 UTC m=+2.116377056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7513a562-0c6a-4272-a1b9-030f19d3600d-iptables-alerter-script\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.666894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-konnectivity-ca\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-host\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4s9\" (UniqueName: \"kubernetes.io/projected/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-kube-api-access-pf4s9\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.666951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-host\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-conf-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-kubernetes\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-sys\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6tq\" (UniqueName: \"kubernetes.io/projected/0ef9acfc-b406-4597-b256-061812bcd27d-kube-api-access-2x6tq\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjtrr\" (UniqueName: \"kubernetes.io/projected/9c0a1341-9866-4678-acf3-753b34a41424-kube-api-access-bjtrr\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-kubelet\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-env-overrides\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-sys\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4n6\" (UniqueName: \"kubernetes.io/projected/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-kube-api-access-sj4n6\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2821554a-7a86-463f-a3bf-78c92fd9cba0-serviceca\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-conf\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.667402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-systemd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-var-lib-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-config\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-tmp\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-socket-dir-parent\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-systemd-units\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-bin\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysconfig\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7513a562-0c6a-4272-a1b9-030f19d3600d-host-slash\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667605    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-konnectivity-ca\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667662    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-kubelet-config\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2821554a-7a86-463f-a3bf-78c92fd9cba0-host\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-kubelet\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.668292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667758    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-slash\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-etc-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-system-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-multus\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs4b\" (UniqueName: \"kubernetes.io/projected/2821554a-7a86-463f-a3bf-78c92fd9cba0-kube-api-access-jjs4b\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7513a562-0c6a-4272-a1b9-030f19d3600d-iptables-alerter-script\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-lib-modules\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-slash\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-kubelet-config\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-k8s-cni-cncf-io\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.667996    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-env-overrides\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668012    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2821554a-7a86-463f-a3bf-78c92fd9cba0-host\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-k8s-cni-cncf-io\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-cni-bin\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-hostroot\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-socket-dir-parent\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-etc-kubernetes\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-systemd-units\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-netns\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-lib-modules\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668131    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-log-socket\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-etc-kubernetes\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d07d2f8-cd59-4b61-8b13-27675f985d5b-tmp-dir\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-multus-conf-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-system-cni-dir\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668210    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-etc-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-hostroot\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-var-lib-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-kubernetes\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.669949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-netns\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysctl-conf\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-ovn\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-host-run-netns\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-netns\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovn-node-metrics-cert\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-systemd\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-cnibin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-multus-daemon-config\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-multus-certs\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-dbus\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjfc\" (UniqueName: \"kubernetes.io/projected/7513a562-0c6a-4272-a1b9-030f19d3600d-kube-api-access-dhjfc\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cni-binary-copy\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-system-cni-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cnibin\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-log-socket\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d07d2f8-cd59-4b61-8b13-27675f985d5b-hosts-file\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668672    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-cni-binary-copy\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.670874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-modprobe-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ln6\" (UniqueName: \"kubernetes.io/projected/6d07d2f8-cd59-4b61-8b13-27675f985d5b-kube-api-access-25ln6\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-ovn\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668864    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-cnibin\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2821554a-7a86-463f-a3bf-78c92fd9cba0-serviceca\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-systemd\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-system-cni-dir\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668926    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-sysconfig\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-systemd\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-run-openvswitch\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-cnibin\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-var-lib-cni-multus\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9c0a1341-9866-4678-acf3-753b34a41424-host-run-multus-certs\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d07d2f8-cd59-4b61-8b13-27675f985d5b-hosts-file\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7513a562-0c6a-4272-a1b9-030f19d3600d-host-slash\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669112    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ef9acfc-b406-4597-b256-061812bcd27d-etc-modprobe-d\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.671547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-etc-tuned\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.668542    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d07d2f8-cd59-4b61-8b13-27675f985d5b-tmp-dir\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991f1b5b-2996-488b-9cc9-45ede6430c6f-dbus\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669653    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-multus-daemon-config\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0a1341-9866-4678-acf3-753b34a41424-cni-binary-copy\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-config\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.669775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovnkube-script-lib\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.670023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bf12ff35-866d-4c47-ab43-d5c2a7946d5a-agent-certs\") pod \"konnectivity-agent-hqpjn\" (UID: \"bf12ff35-866d-4c47-ab43-d5c2a7946d5a\") " pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.670563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ef9acfc-b406-4597-b256-061812bcd27d-tmp\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.672035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.671378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-ovn-node-metrics-cert\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.676196 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.676179    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:41.676196 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.676200    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:41.676406 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.676211    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:41.676406 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:41.676257    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:42.176245094 +0000 UTC m=+2.125825923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:41.677069 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.677042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4n6\" (UniqueName: \"kubernetes.io/projected/60331c31-f2d3-4dff-9ff9-b204a72ff1fc-kube-api-access-sj4n6\") pod \"ovnkube-node-bxkn8\" (UID: \"60331c31-f2d3-4dff-9ff9-b204a72ff1fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.677672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.677649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4s9\" (UniqueName: \"kubernetes.io/projected/cd6a48e2-28d7-4fa6-b143-75a8f7c11858-kube-api-access-pf4s9\") pod \"multus-additional-cni-plugins-mqmsm\" (UID: \"cd6a48e2-28d7-4fa6-b143-75a8f7c11858\") " pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.678316 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.678300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6tq\" (UniqueName: \"kubernetes.io/projected/0ef9acfc-b406-4597-b256-061812bcd27d-kube-api-access-2x6tq\") pod \"tuned-8jff9\" (UID: \"0ef9acfc-b406-4597-b256-061812bcd27d\") " pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.678381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.678320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjtrr\" (UniqueName: \"kubernetes.io/projected/9c0a1341-9866-4678-acf3-753b34a41424-kube-api-access-bjtrr\") pod \"multus-cnfqg\" (UID: \"9c0a1341-9866-4678-acf3-753b34a41424\") " pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.678858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.678833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ln6\" (UniqueName: \"kubernetes.io/projected/6d07d2f8-cd59-4b61-8b13-27675f985d5b-kube-api-access-25ln6\") pod \"node-resolver-cm49p\" (UID: \"6d07d2f8-cd59-4b61-8b13-27675f985d5b\") " pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.678858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.678855    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs4b\" (UniqueName: \"kubernetes.io/projected/2821554a-7a86-463f-a3bf-78c92fd9cba0-kube-api-access-jjs4b\") pod \"node-ca-9w9h5\" (UID: \"2821554a-7a86-463f-a3bf-78c92fd9cba0\") " pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.678981 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.678951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjfc\" (UniqueName: \"kubernetes.io/projected/7513a562-0c6a-4272-a1b9-030f19d3600d-kube-api-access-dhjfc\") pod \"iptables-alerter-mjkz5\" (UID: \"7513a562-0c6a-4272-a1b9-030f19d3600d\") " pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.729333 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.729300    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb996e25865c7c90759e7b27b65ff0d4.slice/crio-45fd0bf44a599741bd7a6b4a5f1374202048b674ac3c04dcf0bcb86db3ff666d WatchSource:0}: Error finding container 45fd0bf44a599741bd7a6b4a5f1374202048b674ac3c04dcf0bcb86db3ff666d: Status 404 returned error can't find the container with id 45fd0bf44a599741bd7a6b4a5f1374202048b674ac3c04dcf0bcb86db3ff666d
Sep 09 16:13:41.739384 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.739361    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:41.787713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.787645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct"
Sep 09 16:13:41.795856 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.795834    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd64f617_1030_43b9_bf58_a4e01eaed827.slice/crio-c7f2c569cf28fdaa339c19277b24678acbb01c4fec33f19f5adfc1aeb885a364 WatchSource:0}: Error finding container c7f2c569cf28fdaa339c19277b24678acbb01c4fec33f19f5adfc1aeb885a364: Status 404 returned error can't find the container with id c7f2c569cf28fdaa339c19277b24678acbb01c4fec33f19f5adfc1aeb885a364
Sep 09 16:13:41.802953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.802936    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jff9"
Sep 09 16:13:41.810360 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.810337    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef9acfc_b406_4597_b256_061812bcd27d.slice/crio-39b864e1d656cb07dca834482b31246513a68aba69cf09fe4627e41a2b5cc625 WatchSource:0}: Error finding container 39b864e1d656cb07dca834482b31246513a68aba69cf09fe4627e41a2b5cc625: Status 404 returned error can't find the container with id 39b864e1d656cb07dca834482b31246513a68aba69cf09fe4627e41a2b5cc625
Sep 09 16:13:41.817994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.817974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mjkz5"
Sep 09 16:13:41.825662 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.825643    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7513a562_0c6a_4272_a1b9_030f19d3600d.slice/crio-2cfd844943d7454d4d0e7b89590573d9cbb6e41f71b1d070afdf83a738f3a20b WatchSource:0}: Error finding container 2cfd844943d7454d4d0e7b89590573d9cbb6e41f71b1d070afdf83a738f3a20b: Status 404 returned error can't find the container with id 2cfd844943d7454d4d0e7b89590573d9cbb6e41f71b1d070afdf83a738f3a20b
Sep 09 16:13:41.837006 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.836935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mqmsm"
Sep 09 16:13:41.845094 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.845056    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6a48e2_28d7_4fa6_b143_75a8f7c11858.slice/crio-de5b91680a1e8580bed74d3cc99e674cba76c427982fb402148ab10b5fd9fb2d WatchSource:0}: Error finding container de5b91680a1e8580bed74d3cc99e674cba76c427982fb402148ab10b5fd9fb2d: Status 404 returned error can't find the container with id de5b91680a1e8580bed74d3cc99e674cba76c427982fb402148ab10b5fd9fb2d
Sep 09 16:13:41.850478 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.850461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cnfqg"
Sep 09 16:13:41.857616 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.857596    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0a1341_9866_4678_acf3_753b34a41424.slice/crio-92a73dd3cb11b77b4350b4a126f0f059941f6e4d40b622ae33526dc8a3bbc648 WatchSource:0}: Error finding container 92a73dd3cb11b77b4350b4a126f0f059941f6e4d40b622ae33526dc8a3bbc648: Status 404 returned error can't find the container with id 92a73dd3cb11b77b4350b4a126f0f059941f6e4d40b622ae33526dc8a3bbc648
Sep 09 16:13:41.859379 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.859364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:13:41.866459 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.866440    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf12ff35_866d_4c47_ab43_d5c2a7946d5a.slice/crio-e379bb865751b455cd567f5ddf53453ed212c8a09f817016385dcfc7889f78f0 WatchSource:0}: Error finding container e379bb865751b455cd567f5ddf53453ed212c8a09f817016385dcfc7889f78f0: Status 404 returned error can't find the container with id e379bb865751b455cd567f5ddf53453ed212c8a09f817016385dcfc7889f78f0
Sep 09 16:13:41.867144 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.867130    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:13:41.872863 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.872847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cm49p"
Sep 09 16:13:41.875228 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.875207    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60331c31_f2d3_4dff_9ff9_b204a72ff1fc.slice/crio-3df5b57132cbf8b26178fff4aa230713b1caf9d1fececd173b003c47a037ca47 WatchSource:0}: Error finding container 3df5b57132cbf8b26178fff4aa230713b1caf9d1fececd173b003c47a037ca47: Status 404 returned error can't find the container with id 3df5b57132cbf8b26178fff4aa230713b1caf9d1fececd173b003c47a037ca47
Sep 09 16:13:41.881091 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:41.881073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9w9h5"
Sep 09 16:13:41.889217 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:13:41.889194    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2821554a_7a86_463f_a3bf_78c92fd9cba0.slice/crio-3b84b93bd6d6a2c7d0fbaa1dd49c7f8ebeebd2635195dac1883353006b638591 WatchSource:0}: Error finding container 3b84b93bd6d6a2c7d0fbaa1dd49c7f8ebeebd2635195dac1883353006b638591: Status 404 returned error can't find the container with id 3b84b93bd6d6a2c7d0fbaa1dd49c7f8ebeebd2635195dac1883353006b638591
Sep 09 16:13:42.072090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.072012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:42.072220 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.072114    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:42.072220 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.072158    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:43.072145711 +0000 UTC m=+3.021726535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:42.172316 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.172287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:42.172516 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.172427    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:42.172516 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.172497    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:43.172477677 +0000 UTC m=+3.122058513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:42.273077 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.273043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:42.273280 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.273198    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:42.273280 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.273217    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:42.273280 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.273228    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:42.273428 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:42.273289    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:43.273272361 +0000 UTC m=+3.222853188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:42.497142 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.497028    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:08:41 +0000 UTC" deadline="2028-05-15 05:11:16.845674345 +0000 UTC"
Sep 09 16:13:42.497142 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.497090    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14724h57m34.34858787s"
Sep 09 16:13:42.534431 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.534342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jff9" event={"ID":"0ef9acfc-b406-4597-b256-061812bcd27d","Type":"ContainerStarted","Data":"39b864e1d656cb07dca834482b31246513a68aba69cf09fe4627e41a2b5cc625"}
Sep 09 16:13:42.537080 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.537050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct" event={"ID":"dd64f617-1030-43b9-bf58-a4e01eaed827","Type":"ContainerStarted","Data":"c7f2c569cf28fdaa339c19277b24678acbb01c4fec33f19f5adfc1aeb885a364"}
Sep 09 16:13:42.540427 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.540311    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal" event={"ID":"92a982754da1ab91809fe032c16435bd","Type":"ContainerStarted","Data":"b1bac193c8025016c3716f6f0b45c77d935120da63c5d2bce98f24668c289f6a"}
Sep 09 16:13:42.543226 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.543116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w9h5" event={"ID":"2821554a-7a86-463f-a3bf-78c92fd9cba0","Type":"ContainerStarted","Data":"3b84b93bd6d6a2c7d0fbaa1dd49c7f8ebeebd2635195dac1883353006b638591"}
Sep 09 16:13:42.545811 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.545787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnfqg" event={"ID":"9c0a1341-9866-4678-acf3-753b34a41424","Type":"ContainerStarted","Data":"92a73dd3cb11b77b4350b4a126f0f059941f6e4d40b622ae33526dc8a3bbc648"}
Sep 09 16:13:42.547701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.547675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjkz5" event={"ID":"7513a562-0c6a-4272-a1b9-030f19d3600d","Type":"ContainerStarted","Data":"2cfd844943d7454d4d0e7b89590573d9cbb6e41f71b1d070afdf83a738f3a20b"}
Sep 09 16:13:42.551413 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.551388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal" event={"ID":"db996e25865c7c90759e7b27b65ff0d4","Type":"ContainerStarted","Data":"45fd0bf44a599741bd7a6b4a5f1374202048b674ac3c04dcf0bcb86db3ff666d"}
Sep 09 16:13:42.553857 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.553790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cm49p" event={"ID":"6d07d2f8-cd59-4b61-8b13-27675f985d5b","Type":"ContainerStarted","Data":"c04a399cede3768ea80ee8e1833efee4eef66ee430eaef8cfaf33a25578d0764"}
Sep 09 16:13:42.555416 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.555393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"3df5b57132cbf8b26178fff4aa230713b1caf9d1fececd173b003c47a037ca47"}
Sep 09 16:13:42.563737 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.563713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hqpjn" event={"ID":"bf12ff35-866d-4c47-ab43-d5c2a7946d5a","Type":"ContainerStarted","Data":"e379bb865751b455cd567f5ddf53453ed212c8a09f817016385dcfc7889f78f0"}
Sep 09 16:13:42.567301 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.567237    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerStarted","Data":"de5b91680a1e8580bed74d3cc99e674cba76c427982fb402148ab10b5fd9fb2d"}
Sep 09 16:13:42.617164 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.616976    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:42.779783 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:42.779708    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:43.080722 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.080649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:43.080878 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.080806    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:43.080937 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.080878    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:45.080857265 +0000 UTC m=+5.030438102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:43.181764 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.181724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:43.181942 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.181888    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:43.182050 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.181947    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:45.18193007 +0000 UTC m=+5.131510899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:43.283160 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.283123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:43.283336 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.283283    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:43.283336 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.283303    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:43.283336 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.283315    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:43.283492 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.283371    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:45.283353357 +0000 UTC m=+5.232934186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:43.498221 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.498136    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 16:08:41 +0000 UTC" deadline="2028-04-09 07:42:46.704031891 +0000 UTC"
Sep 09 16:13:43.498221 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.498177    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13863h29m3.205860032s"
Sep 09 16:13:43.526008 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.525981    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:43.526165 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.526118    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:43.526517 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.526499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:43.526623 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.526605    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:43.526675 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:43.526666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:43.526810 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:43.526792    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:44.474231 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:44.474199    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 16:13:45.098822 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.098783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:45.099332 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.098936    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:45.099332 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.099002    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:49.098987259 +0000 UTC m=+9.048568101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:45.199119 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.199083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:45.199299 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.199286    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:45.199363 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.199339    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:49.199324705 +0000 UTC m=+9.148905529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:45.299537 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.299499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:45.299701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.299640    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:45.299701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.299654    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:45.299701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.299662    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:45.299858 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.299710    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:49.299697632 +0000 UTC m=+9.249278456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:45.525778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.525698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:45.525778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.525714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:45.525974 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:45.525704    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:45.525974 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.525803    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:45.525974 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.525925    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:45.526106 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:45.526012    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:47.525827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:47.525954    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:47.526322    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:47.526430    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:47.526475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:47.526584 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:47.526545    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:49.129749 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.129491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:49.129749 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.129642    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:49.129749 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.129702    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.129681901 +0000 UTC m=+17.079262745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:49.230290 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.230250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:49.230436 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.230396    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:49.230492 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.230471    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.230448333 +0000 UTC m=+17.180029160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:49.331656 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.331575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:49.331803 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.331754    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:49.331803 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.331772    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:49.331803 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.331786    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:49.332234 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.332216    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:13:57.331827307 +0000 UTC m=+17.281408134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:49.525827 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.525794    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:49.526035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.525837    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:49.526035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:49.525800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:49.526035 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.525919    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:49.526035 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.526007    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:49.526240 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:49.526082    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:51.525681 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:51.525643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:51.526185 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:51.525658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:51.526185 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:51.525781    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:51.526185 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:51.525774    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:51.526185 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:51.525893    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:51.526185 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:51.526003    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:53.525701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:53.525670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:53.526086 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:53.525676    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:53.526086 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:53.525787    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:53.526086 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:53.525874    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:53.526086 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:53.525678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:53.526086 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:53.525974    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:55.525214 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:55.525181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:55.525639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:55.525220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:55.525639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:55.525292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:55.525639 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:55.525317    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:55.525639 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:55.525368    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:55.525639 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:55.525456    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:57.184261 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.184227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:57.184746 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.184369    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:57.184746 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.184437    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:13.184418081 +0000 UTC m=+33.133998907 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:13:57.285318 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.285281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:57.285491 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.285401    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:57.285491 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.285470    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret podName:991f1b5b-2996-488b-9cc9-45ede6430c6f nodeName:}" failed. No retries permitted until 2026-09-09 16:14:13.285450902 +0000 UTC m=+33.235031740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret") pod "global-pull-secret-syncer-5btpt" (UID: "991f1b5b-2996-488b-9cc9-45ede6430c6f") : object "kube-system"/"original-pull-secret" not registered
Sep 09 16:13:57.385844 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.385809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:57.386057 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.386007    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 16:13:57.386057 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.386033    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 16:13:57.386057 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.386048    2559 projected.go:196] Error preparing data for projected volume kube-api-access-qnpmf for pod openshift-network-diagnostics/network-check-target-f5tf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:57.386221 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.386108    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf podName:f27803b5-592e-4448-94d5-7009bdb04824 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:13.386090872 +0000 UTC m=+33.335671699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnpmf" (UniqueName: "kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf") pod "network-check-target-f5tf2" (UID: "f27803b5-592e-4448-94d5-7009bdb04824") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 16:13:57.525290 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.525211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:57.525455 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.525339    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:57.525455 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.525215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:57.525455 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.525441    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:13:57.525455 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:57.525211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:57.525647 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:57.525511    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:59.525431 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:59.525402    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:13:59.525761 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:59.525402    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:13:59.525761 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:59.525498    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:13:59.525761 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:59.525562    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:13:59.525761 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:13:59.525409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:13:59.525761 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:13:59.525659    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:00.598471 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.598263    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cm49p" event={"ID":"6d07d2f8-cd59-4b61-8b13-27675f985d5b","Type":"ContainerStarted","Data":"593d757f2ade3f5f23bfd137aee276f91a910c62ef95653770b7de7ec98a5d8c"}
Sep 09 16:14:00.601189 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601170    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:14:00.601481 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601464    2559 generic.go:356] "Generic (PLEG): container finished" podID="60331c31-f2d3-4dff-9ff9-b204a72ff1fc" containerID="bcf72eda0e0ab6ae0a7305133bf4d5be4bbe3a34ca82964bbd619cc1a3df805a" exitCode=1
Sep 09 16:14:00.601564 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"053770d1118bf3ca6189aa238b20f1dab3d5cefd45f4091c1c8536337dccbbbc"}
Sep 09 16:14:00.601607 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"9b1e7ec784381fa36e54e4d18d2e0cf14484ab0c8542c67361ce84a1d9acc8a2"}
Sep 09 16:14:00.601659 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"7615478460e45a015d50f8810eb893d2f87cfeca33b6906d97ced104f4e59c30"}
Sep 09 16:14:00.601712 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"61d1a147be3e7b66dbf0e9ae2dd4a9b85e8ae1a163b0f3ae3b06c02452514cb3"}
Sep 09 16:14:00.601757 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerDied","Data":"bcf72eda0e0ab6ae0a7305133bf4d5be4bbe3a34ca82964bbd619cc1a3df805a"}
Sep 09 16:14:00.601804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.601771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"faf29b7df3efe2ae291d904d7176c4fbddb9404ac73ae7d5d759b6f9f61e2d8e"}
Sep 09 16:14:00.602839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.602815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hqpjn" event={"ID":"bf12ff35-866d-4c47-ab43-d5c2a7946d5a","Type":"ContainerStarted","Data":"31353e04b0d3e02539ab0902fb9f169065b12736519f8c54f4cd28671a83d43a"}
Sep 09 16:14:00.604206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.604188    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="6e12fae38c1ce745b133be065f72d426dae7be7edaba9157267bdedfacfef7a0" exitCode=0
Sep 09 16:14:00.604270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.604243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"6e12fae38c1ce745b133be065f72d426dae7be7edaba9157267bdedfacfef7a0"}
Sep 09 16:14:00.605647 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.605626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jff9" event={"ID":"0ef9acfc-b406-4597-b256-061812bcd27d","Type":"ContainerStarted","Data":"60921be1d16cfffe685a4d0d44a9488c22d9853e25746985e065746ad0a2d9f1"}
Sep 09 16:14:00.606982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.606946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct" event={"ID":"dd64f617-1030-43b9-bf58-a4e01eaed827","Type":"ContainerStarted","Data":"38257af545b14325ab6ae93703b5ee97bedc11584cafd06a68dc772300f7a303"}
Sep 09 16:14:00.607778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.607762    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:14:00.608376 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.608359    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal" event={"ID":"92a982754da1ab91809fe032c16435bd","Type":"ContainerStarted","Data":"f044311ee819cc1aa406dbb2c00c86e37401f658c75f2e18eb7afeab870239bd"}
Sep 09 16:14:00.608781 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.608765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:14:00.609748 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.609730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9w9h5" event={"ID":"2821554a-7a86-463f-a3bf-78c92fd9cba0","Type":"ContainerStarted","Data":"7b7526348a6102c264e5d8a8a91d573a4ab325e1fdaaf49e5d69450cd6c3ea7f"}
Sep 09 16:14:00.611053 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.611033    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cnfqg" event={"ID":"9c0a1341-9866-4678-acf3-753b34a41424","Type":"ContainerStarted","Data":"77dfc11f1709ac7d738909ed4989eadef3d0c8b58df8867d02940a9efabe71f4"}
Sep 09 16:14:00.612280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.612263    2559 generic.go:356] "Generic (PLEG): container finished" podID="db996e25865c7c90759e7b27b65ff0d4" containerID="044ee51387f6af2ca788a7e154e7a0fef8370c7756ad2856e50b550d850b54eb" exitCode=0
Sep 09 16:14:00.612343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.612287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal" event={"ID":"db996e25865c7c90759e7b27b65ff0d4","Type":"ContainerDied","Data":"044ee51387f6af2ca788a7e154e7a0fef8370c7756ad2856e50b550d850b54eb"}
Sep 09 16:14:00.613230 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.613197    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cm49p" podStartSLOduration=6.708071808 podStartE2EDuration="20.61318527s" totalImagesPullingTime="13.905113462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.61287288 +0000 UTC m=+20.562453725" watchObservedRunningTime="2026-09-09 16:14:00.61318527 +0000 UTC m=+20.562766116"
Sep 09 16:14:00.647393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.647360    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cnfqg" podStartSLOduration=2.372599829 podStartE2EDuration="20.647349718s" totalImagesPullingTime="18.274749889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.646992235 +0000 UTC m=+20.596573081" watchObservedRunningTime="2026-09-09 16:14:00.647349718 +0000 UTC m=+20.596930563"
Sep 09 16:14:00.661344 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.661307    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9w9h5" podStartSLOduration=10.315961896 podStartE2EDuration="20.661297555s" totalImagesPullingTime="10.345335659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.66123679 +0000 UTC m=+20.610817636" watchObservedRunningTime="2026-09-09 16:14:00.661297555 +0000 UTC m=+20.610878400"
Sep 09 16:14:00.676246 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.676206    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-104.ec2.internal" podStartSLOduration=19.676197336 podStartE2EDuration="19.676197336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.675741433 +0000 UTC m=+20.625322279" watchObservedRunningTime="2026-09-09 16:14:00.676197336 +0000 UTC m=+20.625778179"
Sep 09 16:14:00.742357 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.742279    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8jff9" podStartSLOduration=10.318256649 podStartE2EDuration="20.742265394s" totalImagesPullingTime="10.424008745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.707240796 +0000 UTC m=+20.656821641" watchObservedRunningTime="2026-09-09 16:14:00.742265394 +0000 UTC m=+20.691846236"
Sep 09 16:14:00.742516 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:00.742498    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hqpjn" podStartSLOduration=12.987428914 podStartE2EDuration="20.742492706s" totalImagesPullingTime="7.755063792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:00.742153785 +0000 UTC m=+20.691734629" watchObservedRunningTime="2026-09-09 16:14:00.742492706 +0000 UTC m=+20.692073590"
Sep 09 16:14:01.241689 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.241665    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 16:14:01.525529 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.525501    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:01.525701 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:01.525610    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:01.525762 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.525716    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:01.525854 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:01.525833    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:01.525904 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.525862    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:01.525934 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:01.525919    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:01.527597 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.527534    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T16:14:01.241684065Z","UUID":"6fcb8298-3ad9-41a4-8c64-c17babb440ef","Handler":null,"Name":"","Endpoint":""}
Sep 09 16:14:01.528793 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.528777    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 16:14:01.528868 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.528800    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 16:14:01.615877 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.615851    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct" event={"ID":"dd64f617-1030-43b9-bf58-a4e01eaed827","Type":"ContainerStarted","Data":"dcbb7e64e8816baed8ff5a1bec404096f215035b85c7aad8c66704e15a5c60dc"}
Sep 09 16:14:01.617323 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.617292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mjkz5" event={"ID":"7513a562-0c6a-4272-a1b9-030f19d3600d","Type":"ContainerStarted","Data":"51581af422ec546f932bd3c08a8f58cde7722fe0b0eb570bbed003c1051b6d67"}
Sep 09 16:14:01.617614 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.617592    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:14:01.618449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.618022    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hqpjn"
Sep 09 16:14:01.630868 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:01.630830    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mjkz5" podStartSLOduration=7.669811391 podStartE2EDuration="21.630819115s" totalImagesPullingTime="13.961007724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:01.630621752 +0000 UTC m=+21.580202598" watchObservedRunningTime="2026-09-09 16:14:01.630819115 +0000 UTC m=+21.580399961"
Sep 09 16:14:02.621809 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.621773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct" event={"ID":"dd64f617-1030-43b9-bf58-a4e01eaed827","Type":"ContainerStarted","Data":"e0732499ead8dda31b68245e159e8326a2ea6dec9a0824645ad6bee6f0a833c5"}
Sep 09 16:14:02.626000 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.625972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal" event={"ID":"db996e25865c7c90759e7b27b65ff0d4","Type":"ContainerStarted","Data":"7d3d571d9dc7b1c723d18ce036dfdfb43861b25377207f01fc24aed0e7362f86"}
Sep 09 16:14:02.628806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.628789    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:14:02.629107 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.629086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"6f5b891830b25f536bb3f52353352940586e47c839332ca13f753400f57e1bc3"}
Sep 09 16:14:02.640320 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.640285    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlsct" podStartSLOduration=7.016498554 podStartE2EDuration="22.640273137s" totalImagesPullingTime="15.623774583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:02.639011732 +0000 UTC m=+22.588592577" watchObservedRunningTime="2026-09-09 16:14:02.640273137 +0000 UTC m=+22.589853983"
Sep 09 16:14:02.652413 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:02.652380    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-104.ec2.internal" podStartSLOduration=21.652368967 podStartE2EDuration="21.652368967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:02.652240221 +0000 UTC m=+22.601821066" watchObservedRunningTime="2026-09-09 16:14:02.652368967 +0000 UTC m=+22.601949812"
Sep 09 16:14:03.525203 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:03.525174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:03.525392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:03.525175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:03.525392 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:03.525262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:03.525392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:03.525178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:03.525392 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:03.525339    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:03.525532 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:03.525393    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:04.637418 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:04.637256    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:14:04.637999 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:04.637971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"cb28d47b86504d252982581831745a005e4f096c03548ad223e1605be6b0bfb1"}
Sep 09 16:14:04.638371 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:04.638237    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:04.638493 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:04.638427    2559 scope.go:117] "RemoveContainer" containerID="bcf72eda0e0ab6ae0a7305133bf4d5be4bbe3a34ca82964bbd619cc1a3df805a"
Sep 09 16:14:04.657584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:04.657559    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:05.525544 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.525517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:05.525704 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.525517    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:05.525704 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:05.525641    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:05.525704 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.525518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:05.525863 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:05.525746    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:05.525863 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:05.525777    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:05.644631 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.644600    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:14:05.645193 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.645029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" event={"ID":"60331c31-f2d3-4dff-9ff9-b204a72ff1fc","Type":"ContainerStarted","Data":"ec128ca4516e5eb3ace7ca6f5ed8f1c45e52e9a7ecbae3026af94ffaf49cfa99"}
Sep 09 16:14:05.645274 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.645243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:05.645383 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.645322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:05.661971 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.661940    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:05.673304 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:05.673259    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8" podStartSLOduration=7.898493313 podStartE2EDuration="25.673242369s" totalImagesPullingTime="17.774749056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:05.672803499 +0000 UTC m=+25.622384346" watchObservedRunningTime="2026-09-09 16:14:05.673242369 +0000 UTC m=+25.622823216"
Sep 09 16:14:06.439513 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.439483    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5tf2"]
Sep 09 16:14:06.439648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.439590    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:06.439692 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:06.439673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:06.442980 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.442909    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5btpt"]
Sep 09 16:14:06.443197 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.443147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:06.443894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.443618    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgf65"]
Sep 09 16:14:06.444322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:06.444290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:06.444411 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:06.443899    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:06.444501 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:06.444461    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:07.525233 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:07.525205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:07.525920 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:07.525319    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:08.525193 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:08.525161    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:08.525356 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:08.525164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:08.525356 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:08.525297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:08.525757 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:08.525372    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:09.525333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:09.525170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:09.525471 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:09.525408    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:09.657400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:09.657366    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="fffb8c2fcc23ea92ad0c22e968b75cced0b6e854186e7729fbc0ec405f398027" exitCode=0
Sep 09 16:14:09.657530 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:09.657418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"fffb8c2fcc23ea92ad0c22e968b75cced0b6e854186e7729fbc0ec405f398027"}
Sep 09 16:14:10.525843 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:10.525815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:10.526237 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:10.525890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:10.526237 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:10.525903    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5tf2" podUID="f27803b5-592e-4448-94d5-7009bdb04824"
Sep 09 16:14:10.526237 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:10.525988    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5btpt" podUID="991f1b5b-2996-488b-9cc9-45ede6430c6f"
Sep 09 16:14:10.664466 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:10.664386    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="a34434c0b1cc5f04504743ae8a9f80068df0937e7fcb3d414d9430c3bef7f563" exitCode=0
Sep 09 16:14:10.664466 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:10.664429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"a34434c0b1cc5f04504743ae8a9f80068df0937e7fcb3d414d9430c3bef7f563"}
Sep 09 16:14:11.525943 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.525914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:11.526278 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:11.526029    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:14:11.669179 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.669116    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="07eba222f9d3441f65d807c66dfa3b0bf47217ecadcd49757fee3d043c780bf6" exitCode=0
Sep 09 16:14:11.669179 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.669157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"07eba222f9d3441f65d807c66dfa3b0bf47217ecadcd49757fee3d043c780bf6"}
Sep 09 16:14:11.859014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.858988    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-104.ec2.internal" event="NodeReady"
Sep 09 16:14:11.859114 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.859075    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 16:14:11.944811 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.944747    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bhfmc"]
Sep 09 16:14:11.965051 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.965027    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5x9bq"]
Sep 09 16:14:11.965163 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.965152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:11.967865 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.967841    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vg786\""
Sep 09 16:14:11.968231 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.968211    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 16:14:11.968583 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.968555    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 16:14:11.968693 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.968600    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 16:14:11.976451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.976432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhfmc"]
Sep 09 16:14:11.976550 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.976458    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x9bq"]
Sep 09 16:14:11.976611 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.976552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:11.979765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.979748    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 16:14:11.979765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.979762    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjfwh\""
Sep 09 16:14:11.979864 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:11.979795    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 16:14:12.092505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.092628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-tmp-dir\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.092628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmqf\" (UniqueName: \"kubernetes.io/projected/9f61d873-8851-48cb-9b3d-5243316b65a8-kube-api-access-6kmqf\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.092628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f61d873-8851-48cb-9b3d-5243316b65a8-cert\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.092628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-config-volume\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.092769 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.092629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrsg\" (UniqueName: \"kubernetes.io/projected/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-kube-api-access-gdrsg\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.193763 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmqf\" (UniqueName: \"kubernetes.io/projected/9f61d873-8851-48cb-9b3d-5243316b65a8-kube-api-access-6kmqf\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.193886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f61d873-8851-48cb-9b3d-5243316b65a8-cert\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.193886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-config-volume\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.193886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrsg\" (UniqueName: \"kubernetes.io/projected/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-kube-api-access-gdrsg\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.193886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.193886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.193866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-tmp-dir\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.194126 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:12.194024    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:12.194126 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:12.194095    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:12.694075802 +0000 UTC m=+32.643656645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:12.194374 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.194355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-tmp-dir\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.194615 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.194574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-config-volume\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.197850 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.197722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f61d873-8851-48cb-9b3d-5243316b65a8-cert\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.202886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.202860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmqf\" (UniqueName: \"kubernetes.io/projected/9f61d873-8851-48cb-9b3d-5243316b65a8-kube-api-access-6kmqf\") pod \"ingress-canary-bhfmc\" (UID: \"9f61d873-8851-48cb-9b3d-5243316b65a8\") " pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.203163 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.203144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrsg\" (UniqueName: \"kubernetes.io/projected/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-kube-api-access-gdrsg\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.274629 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.274610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhfmc"
Sep 09 16:14:12.528645 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.528576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:12.529165 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.528821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:12.531140 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.531117    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 16:14:12.531269 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.531146    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 16:14:12.531269 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.531190    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 16:14:12.531363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.531321    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7wfvc\""
Sep 09 16:14:12.580296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.580267    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhfmc"]
Sep 09 16:14:12.584453 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:14:12.584425    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f61d873_8851_48cb_9b3d_5243316b65a8.slice/crio-60d38b0d09cb5c0172921741fef76a48978b7fbe0b57edf0364af803ceefd4b1 WatchSource:0}: Error finding container 60d38b0d09cb5c0172921741fef76a48978b7fbe0b57edf0364af803ceefd4b1: Status 404 returned error can't find the container with id 60d38b0d09cb5c0172921741fef76a48978b7fbe0b57edf0364af803ceefd4b1
Sep 09 16:14:12.672678 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.672642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhfmc" event={"ID":"9f61d873-8851-48cb-9b3d-5243316b65a8","Type":"ContainerStarted","Data":"60d38b0d09cb5c0172921741fef76a48978b7fbe0b57edf0364af803ceefd4b1"}
Sep 09 16:14:12.697955 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:12.697928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:12.698096 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:12.698068    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:12.698144 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:12.698121    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:13.698105992 +0000 UTC m=+33.647686816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:13.201145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.201101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:13.201412 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:13.201227    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:13.201412 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:13.201303    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:45.201286636 +0000 UTC m=+65.150867463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 16:14:13.302433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.302399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:13.305289 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.305256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991f1b5b-2996-488b-9cc9-45ede6430c6f-original-pull-secret\") pod \"global-pull-secret-syncer-5btpt\" (UID: \"991f1b5b-2996-488b-9cc9-45ede6430c6f\") " pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:13.403572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.403540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:13.406346 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.406321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpmf\" (UniqueName: \"kubernetes.io/projected/f27803b5-592e-4448-94d5-7009bdb04824-kube-api-access-qnpmf\") pod \"network-check-target-f5tf2\" (UID: \"f27803b5-592e-4448-94d5-7009bdb04824\") " pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:13.438230 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.438205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:13.443306 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.443284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5btpt"
Sep 09 16:14:13.525415 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.525388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:13.528252 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.528233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:14:13.528376 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.528236    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjzmn\""
Sep 09 16:14:13.706846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.706774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:13.707466 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:13.706906    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:13.707466 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:13.706987    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:15.706951191 +0000 UTC m=+35.656532018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:13.715326 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.715301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5btpt"]
Sep 09 16:14:13.718501 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:13.718478    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5tf2"]
Sep 09 16:14:13.720357 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:14:13.720278    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991f1b5b_2996_488b_9cc9_45ede6430c6f.slice/crio-19f1630a7a1994e3fe5fc689449471785102901d3ccb2f196e6727bfe7859576 WatchSource:0}: Error finding container 19f1630a7a1994e3fe5fc689449471785102901d3ccb2f196e6727bfe7859576: Status 404 returned error can't find the container with id 19f1630a7a1994e3fe5fc689449471785102901d3ccb2f196e6727bfe7859576
Sep 09 16:14:13.722692 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:14:13.722425    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27803b5_592e_4448_94d5_7009bdb04824.slice/crio-b7cc64889f13a5b5350d6b8cba7ac4934f27bc0f88bd84d417fbb08ab109a182 WatchSource:0}: Error finding container b7cc64889f13a5b5350d6b8cba7ac4934f27bc0f88bd84d417fbb08ab109a182: Status 404 returned error can't find the container with id b7cc64889f13a5b5350d6b8cba7ac4934f27bc0f88bd84d417fbb08ab109a182
Sep 09 16:14:14.679193 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:14.679160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5btpt" event={"ID":"991f1b5b-2996-488b-9cc9-45ede6430c6f","Type":"ContainerStarted","Data":"19f1630a7a1994e3fe5fc689449471785102901d3ccb2f196e6727bfe7859576"}
Sep 09 16:14:14.680247 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:14.680229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5tf2" event={"ID":"f27803b5-592e-4448-94d5-7009bdb04824","Type":"ContainerStarted","Data":"b7cc64889f13a5b5350d6b8cba7ac4934f27bc0f88bd84d417fbb08ab109a182"}
Sep 09 16:14:15.685664 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:15.685415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhfmc" event={"ID":"9f61d873-8851-48cb-9b3d-5243316b65a8","Type":"ContainerStarted","Data":"d2ff83bbfc89b4fa09225d4fde0ca873960d299db17d5c53251c435c8d3fd766"}
Sep 09 16:14:15.701715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:15.701631    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bhfmc" podStartSLOduration=2.633745607 podStartE2EDuration="4.701613282s" totalImagesPullingTime="2.067867675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:15.700919868 +0000 UTC m=+35.650500712" watchObservedRunningTime="2026-09-09 16:14:15.701613282 +0000 UTC m=+35.651194129"
Sep 09 16:14:15.720880 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:15.720850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:15.721051 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:15.721001    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:15.721117 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:15.721060    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:19.721043224 +0000 UTC m=+39.670624052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:19.750982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:19.750938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:19.751399 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:19.751110    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:19.751399 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:19.751193    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:27.751166305 +0000 UTC m=+47.700747186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:21.704938 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.704901    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="9867f7403d47da795bcd77ec5c3a10557b556cdbe77d90bd6d6cd9ec5b585d6b" exitCode=0
Sep 09 16:14:21.705329 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.704979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"9867f7403d47da795bcd77ec5c3a10557b556cdbe77d90bd6d6cd9ec5b585d6b"}
Sep 09 16:14:21.706515 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.706489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5btpt" event={"ID":"991f1b5b-2996-488b-9cc9-45ede6430c6f","Type":"ContainerStarted","Data":"1a8dd2211543ba2f666a2207f82244525e254aca5d52c43881efca8527270405"}
Sep 09 16:14:21.708009 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.707987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5tf2" event={"ID":"f27803b5-592e-4448-94d5-7009bdb04824","Type":"ContainerStarted","Data":"93c0a46de0f925a7714c43a0645cc5161c03c020eafe17d8b8bef0737261c4f1"}
Sep 09 16:14:21.708115 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.708105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:14:21.738376 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.738339    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5btpt" podStartSLOduration=34.182693833 podStartE2EDuration="41.738326039s" totalImagesPullingTime="7.555632206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:21.738276658 +0000 UTC m=+41.687857518" watchObservedRunningTime="2026-09-09 16:14:21.738326039 +0000 UTC m=+41.687906886"
Sep 09 16:14:21.753248 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:21.751892    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f5tf2" podStartSLOduration=34.199194558 podStartE2EDuration="41.751876033s" totalImagesPullingTime="7.552681475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:21.751483665 +0000 UTC m=+41.701064512" watchObservedRunningTime="2026-09-09 16:14:21.751876033 +0000 UTC m=+41.701456888"
Sep 09 16:14:22.713559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:22.713529    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd6a48e2-28d7-4fa6-b143-75a8f7c11858" containerID="0cbda1e3d42bf8a1ffcb9ee6cd29b6fd32612e21e01c9ab8134d070b46d0c2ee" exitCode=0
Sep 09 16:14:22.713981 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:22.713618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerDied","Data":"0cbda1e3d42bf8a1ffcb9ee6cd29b6fd32612e21e01c9ab8134d070b46d0c2ee"}
Sep 09 16:14:23.718748 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:23.718713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" event={"ID":"cd6a48e2-28d7-4fa6-b143-75a8f7c11858","Type":"ContainerStarted","Data":"af8f0607d88f2557993fccb3599d39f342e6a4e283ca19ba49c9ff93677a5f7f"}
Sep 09 16:14:23.739798 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:23.739751    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mqmsm" podStartSLOduration=6.450440374 podStartE2EDuration="43.7397365s" totalImagesPullingTime="37.289296126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:14:23.738173269 +0000 UTC m=+43.687754125" watchObservedRunningTime="2026-09-09 16:14:23.7397365 +0000 UTC m=+43.689317345"
Sep 09 16:14:27.806470 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:27.806441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:27.806979 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:27.806555    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:27.806979 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:27.806603    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:14:43.806590339 +0000 UTC m=+63.756171163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:37.662278 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:37.662243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bxkn8"
Sep 09 16:14:43.807834 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:43.807798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:14:43.808241 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:43.807922    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:14:43.808241 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:43.807983    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:15:15.807955699 +0000 UTC m=+95.757536524 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:14:45.216416 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:45.216381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:14:45.218967 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:45.218941    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 16:14:45.226871 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:45.226857    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:14:45.226936 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:14:45.226909    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:15:49.226893831 +0000 UTC m=+129.176474656 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : secret "metrics-daemon-secret" not found
Sep 09 16:14:52.715934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:14:52.715904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f5tf2"
Sep 09 16:15:03.425411 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.425301    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"]
Sep 09 16:15:03.427014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.426999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.429567 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.429541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 16:15:03.429567 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.429556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 16:15:03.429567 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.429566    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 16:15:03.429794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.429620    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 16:15:03.431196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.431178    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 16:15:03.431330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.431259    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 16:15:03.431330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.431325    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 16:15:03.436210 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.436300 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.436300 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.436300 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2712911f-e0fb-49ea-b1bb-a7f59e073645-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.436405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.436405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.436338    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wdp\" (UniqueName: \"kubernetes.io/projected/2712911f-e0fb-49ea-b1bb-a7f59e073645-kube-api-access-t5wdp\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.443129 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.443107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"]
Sep 09 16:15:03.537097 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537097 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2712911f-e0fb-49ea-b1bb-a7f59e073645-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wdp\" (UniqueName: \"kubernetes.io/projected/2712911f-e0fb-49ea-b1bb-a7f59e073645-kube-api-access-t5wdp\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.537901 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.537874    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2712911f-e0fb-49ea-b1bb-a7f59e073645-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.541127 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.541101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.541217 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.541152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-ca\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.541217 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.541162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.541217 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.541181    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2712911f-e0fb-49ea-b1bb-a7f59e073645-hub\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.544790 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.544768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wdp\" (UniqueName: \"kubernetes.io/projected/2712911f-e0fb-49ea-b1bb-a7f59e073645-kube-api-access-t5wdp\") pod \"cluster-proxy-proxy-agent-86476f7b4f-5nm8z\" (UID: \"2712911f-e0fb-49ea-b1bb-a7f59e073645\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:03.749453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:03.749367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:15:04.017895 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:04.017817    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"]
Sep 09 16:15:04.021026 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:15:04.020997    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2712911f_e0fb_49ea_b1bb_a7f59e073645.slice/crio-7f2a0781588d176305720ee0f1cf0152945e3c478c6c7c9dfa21921620c386ff WatchSource:0}: Error finding container 7f2a0781588d176305720ee0f1cf0152945e3c478c6c7c9dfa21921620c386ff: Status 404 returned error can't find the container with id 7f2a0781588d176305720ee0f1cf0152945e3c478c6c7c9dfa21921620c386ff
Sep 09 16:15:04.812774 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:04.812737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerStarted","Data":"7f2a0781588d176305720ee0f1cf0152945e3c478c6c7c9dfa21921620c386ff"}
Sep 09 16:15:06.485547 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:06.485498    2559 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage3229923145/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\": EOF); artifact err: provided artifact is a container image" image="registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02"
Sep 09 16:15:06.485998 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:06.485642    2559 kuberuntime_manager.go:1459] "Unhandled Error" err="container proxy-agent start failed in pod cluster-proxy-proxy-agent-86476f7b4f-5nm8z_open-cluster-management-agent-addon(2712911f-e0fb-49ea-b1bb-a7f59e073645): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage3229923145/2\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\": EOF); artifact err: provided artifact is a container image" logger="UnhandledError"
Sep 09 16:15:06.487294 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:06.487233    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"proxy-agent\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"addon-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"service-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\"]" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645"
Sep 09 16:15:06.820196 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:06.820136    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"proxy-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"addon-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"service-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/multicluster-engine/cluster-proxy-rhel9@sha256:edc2b4f4f42d09008a064ad949a06ee01a18ab696f0691055d29e69927858e02\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage3229923145/2\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/75/75c16de0fb9b68ea6e4534e5d88396228a90b02f99a92692e436fbd32ea69eba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260909T161506Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc4c0898c37fed9238ef39bc0f33fe76cdd126f2a5c17a7faa414abc2e4a5c61&region=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=multicluster-engine----cluster-proxy-rhel9&akamai_signature=exp=1788971406~hmac=8031f0ba77536628fd131e85e3bf7b8313715f90b395578840033472387a422a\\\": EOF); artifact err: provided artifact is a container image\"]" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645"
Sep 09 16:15:15.817977 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:15.817905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:15:15.818347 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:15.818065    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:15:15.818347 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:15.818129    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:16:19.81811319 +0000 UTC m=+159.767694014 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:15:23.857936 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:23.857905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerStarted","Data":"6f67d12fb5bef90d0fe1a44b0002697f70cc1a183877adece47ed39155af8a70"}
Sep 09 16:15:23.858220 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:23.857977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerStarted","Data":"e8c3da05c604b7bb1b90bfeda0fc743bb1bbfa7b8bbfa88a73ee9303a6f73b02"}
Sep 09 16:15:23.858220 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:23.857998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerStarted","Data":"3f85d2055100c1afc5cef20bf00e6b2c19017433f154dd004c53367156a3c88a"}
Sep 09 16:15:23.877427 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:23.877379    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podStartSLOduration=1.19321869 podStartE2EDuration="20.877364141s" totalImagesPullingTime="19.684145451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 16:15:23.876150892 +0000 UTC m=+103.825731737" watchObservedRunningTime="2026-09-09 16:15:23.877364141 +0000 UTC m=+103.826945016"
Sep 09 16:15:49.230906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:15:49.230872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:15:49.231393 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:49.231038    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:15:49.231393 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:15:49.231112    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:17:51.231096618 +0000 UTC m=+251.180677448 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : secret "metrics-daemon-secret" not found
Sep 09 16:16:14.984704 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:16:14.984645    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5x9bq" podUID="cb2d6b57-09e3-438f-be2a-93fcc6aea9e5"
Sep 09 16:16:16.535320 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:16:16.535285    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:16:19.826211 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:16:19.826166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:16:19.826628 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:16:19.826275    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 16:16:19.826628 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:16:19.826330    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls podName:cb2d6b57-09e3-438f-be2a-93fcc6aea9e5 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:21.826313864 +0000 UTC m=+281.775894689 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls") pod "dns-default-5x9bq" (UID: "cb2d6b57-09e3-438f-be2a-93fcc6aea9e5") : secret "dns-default-metrics-tls" not found
Sep 09 16:16:29.525427 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:16:29.525393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:16:31.525685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:16:31.525637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:17:51.233451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:17:51.233412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:17:51.233887 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:17:51.233550    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 16:17:51.233887 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:17:51.233618    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs podName:c6337926-86fe-44e6-82e4-2e7637dc1b70 nodeName:}" failed. No retries permitted until 2026-09-09 16:19:53.233601558 +0000 UTC m=+373.183182389 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs") pod "network-metrics-daemon-mgf65" (UID: "c6337926-86fe-44e6-82e4-2e7637dc1b70") : secret "metrics-daemon-secret" not found
Sep 09 16:18:06.880757 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.880680    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:06.883400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.883384    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.887083 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.887060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8q4lv\""
Sep 09 16:18:06.887203 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.887173    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 16:18:06.887454 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.887439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 16:18:06.888107 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.888093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 16:18:06.894438 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.894419    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 16:18:06.907579 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.907557    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:06.926732 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7xc\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.926831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.926831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.926831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.926831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926823    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.927012 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.927012 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:06.927012 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:06.926890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027711 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7xc\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027711 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.027867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.027950 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.027934    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 16:18:07.028313 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.027986    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9c9565fd-nzxmf: secret "image-registry-tls" not found
Sep 09 16:18:07.028313 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.028071    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls podName:35d36563-9837-400b-bc6c-fa67b73c66f6 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:07.528039283 +0000 UTC m=+267.477620107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls") pod "image-registry-5c9c9565fd-nzxmf" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6") : secret "image-registry-tls" not found
Sep 09 16:18:07.028423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.028348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.028539 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.028516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.028894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.028876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.030214 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.030192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.030284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.030261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.036654 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.036634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7xc\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.036747 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.036731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.531588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:07.531544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:07.531766 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.531691    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 16:18:07.531766 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.531710    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9c9565fd-nzxmf: secret "image-registry-tls" not found
Sep 09 16:18:07.531766 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:07.531766    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls podName:35d36563-9837-400b-bc6c-fa67b73c66f6 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:08.531749899 +0000 UTC m=+268.481330724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls") pod "image-registry-5c9c9565fd-nzxmf" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6") : secret "image-registry-tls" not found
Sep 09 16:18:08.539117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:08.539083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:08.539476 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:08.539222    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 16:18:08.539476 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:08.539239    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9c9565fd-nzxmf: secret "image-registry-tls" not found
Sep 09 16:18:08.539476 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:08.539303    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls podName:35d36563-9837-400b-bc6c-fa67b73c66f6 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:10.539283126 +0000 UTC m=+270.488863959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls") pod "image-registry-5c9c9565fd-nzxmf" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6") : secret "image-registry-tls" not found
Sep 09 16:18:10.551940 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:10.551906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:10.552301 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:10.552053    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 16:18:10.552301 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:10.552066    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9c9565fd-nzxmf: secret "image-registry-tls" not found
Sep 09 16:18:10.552301 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:10.552126    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls podName:35d36563-9837-400b-bc6c-fa67b73c66f6 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:14.552109388 +0000 UTC m=+274.501690224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls") pod "image-registry-5c9c9565fd-nzxmf" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6") : secret "image-registry-tls" not found
Sep 09 16:18:11.384317 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.384286    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-94ss4"]
Sep 09 16:18:11.388301 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.388283    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.390682 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.390661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 16:18:11.391605 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.391585    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 16:18:11.391705 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.391595    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 16:18:11.391705 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.391595    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 16:18:11.391705 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.391595    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q2x6b\""
Sep 09 16:18:11.399677 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.399637    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-94ss4"]
Sep 09 16:18:11.459097 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.459070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44341d7e-e18f-416c-9eee-ecaa4010342b-data-volume\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.459097 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.459097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m66\" (UniqueName: \"kubernetes.io/projected/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-api-access-89m66\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.459270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.459118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44341d7e-e18f-416c-9eee-ecaa4010342b-crio-socket\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.459270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.459163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.459270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.459240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44341d7e-e18f-416c-9eee-ecaa4010342b-data-volume\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m66\" (UniqueName: \"kubernetes.io/projected/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-api-access-89m66\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44341d7e-e18f-416c-9eee-ecaa4010342b-crio-socket\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:11.560233    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44341d7e-e18f-416c-9eee-ecaa4010342b-crio-socket\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:11.560306    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:11.560326    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:12.06030674 +0000 UTC m=+272.009887566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:11.560355    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:12.060343384 +0000 UTC m=+272.009924213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:11.560519 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.560471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44341d7e-e18f-416c-9eee-ecaa4010342b-data-volume\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:11.568067 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:11.568037    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m66\" (UniqueName: \"kubernetes.io/projected/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-api-access-89m66\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:12.063648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:12.063616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:12.063787 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:12.063662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:12.063787 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:12.063738    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:12.063787 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:12.063763    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:12.063787 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:12.063785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:13.063773024 +0000 UTC m=+273.013353862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:12.063935 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:12.063811    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:13.063799027 +0000 UTC m=+273.013379856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:13.072041 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:13.072001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:13.072401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:13.072054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:13.072401 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:13.072120    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:13.072401 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:13.072147    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:13.072401 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:13.072184    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:15.072170397 +0000 UTC m=+275.021751221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:13.072401 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:13.072208    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:15.072195102 +0000 UTC m=+275.021775964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:13.565055 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:13.565022    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cm49p_6d07d2f8-cd59-4b61-8b13-27675f985d5b/dns-node-resolver/0.log"
Sep 09 16:18:14.165979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:14.165932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w9h5_2821554a-7a86-463f-a3bf-78c92fd9cba0/node-ca/0.log"
Sep 09 16:18:14.582803 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:14.582779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:14.582947 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:14.582919    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 16:18:14.582947 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:14.582934    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c9c9565fd-nzxmf: secret "image-registry-tls" not found
Sep 09 16:18:14.583042 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:14.582993    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls podName:35d36563-9837-400b-bc6c-fa67b73c66f6 nodeName:}" failed. No retries permitted until 2026-09-09 16:18:22.582978763 +0000 UTC m=+282.532559587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls") pod "image-registry-5c9c9565fd-nzxmf" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6") : secret "image-registry-tls" not found
Sep 09 16:18:14.965293 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:14.965218    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhfmc_9f61d873-8851-48cb-9b3d-5243316b65a8/serve-healthcheck-canary/0.log"
Sep 09 16:18:15.086154 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:15.086118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:15.086283 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:15.086195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:15.086283 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:15.086248    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:15.086387 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:15.086287    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:15.086387 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:15.086332    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:19.086312081 +0000 UTC m=+279.035892925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:15.086387 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:15.086352    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:19.086341899 +0000 UTC m=+279.035922729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:19.110748 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:19.110703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:19.110748 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:19.110763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:19.111243 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:19.110844    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 16:18:19.111243 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:19.110875    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:19.111243 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:19.110904    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:27.11089001 +0000 UTC m=+287.060470834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : configmap "kube-rbac-proxy" not found
Sep 09 16:18:19.111243 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:19.110939    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:27.110921954 +0000 UTC m=+287.060502782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:21.831407 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:21.831375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:18:21.833575 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:21.833547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb2d6b57-09e3-438f-be2a-93fcc6aea9e5-metrics-tls\") pod \"dns-default-5x9bq\" (UID: \"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5\") " pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:18:22.028786 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.028756    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjfwh\""
Sep 09 16:18:22.036820 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.036804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:18:22.301165 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.301131    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5x9bq"]
Sep 09 16:18:22.304497 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:22.304463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2d6b57_09e3_438f_be2a_93fcc6aea9e5.slice/crio-110e6f901f9a945083766d517d11655cf69c2a40cfb4080aca9b7d77d8573d2f WatchSource:0}: Error finding container 110e6f901f9a945083766d517d11655cf69c2a40cfb4080aca9b7d77d8573d2f: Status 404 returned error can't find the container with id 110e6f901f9a945083766d517d11655cf69c2a40cfb4080aca9b7d77d8573d2f
Sep 09 16:18:22.348138 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.348083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x9bq" event={"ID":"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5","Type":"ContainerStarted","Data":"110e6f901f9a945083766d517d11655cf69c2a40cfb4080aca9b7d77d8573d2f"}
Sep 09 16:18:22.636446 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.636355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:22.638674 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.638645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"image-registry-5c9c9565fd-nzxmf\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") " pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:22.792151 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:22.792118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:23.063770 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:23.063743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:23.066859 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:23.066837    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d36563_9837_400b_bc6c_fa67b73c66f6.slice/crio-947851d46c1571926fb463186cc1371ee1d8f9c33d922513be59b8a9115e48f3 WatchSource:0}: Error finding container 947851d46c1571926fb463186cc1371ee1d8f9c33d922513be59b8a9115e48f3: Status 404 returned error can't find the container with id 947851d46c1571926fb463186cc1371ee1d8f9c33d922513be59b8a9115e48f3
Sep 09 16:18:23.353150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:23.353067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" event={"ID":"35d36563-9837-400b-bc6c-fa67b73c66f6","Type":"ContainerStarted","Data":"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"}
Sep 09 16:18:23.353150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:23.353114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" event={"ID":"35d36563-9837-400b-bc6c-fa67b73c66f6","Type":"ContainerStarted","Data":"947851d46c1571926fb463186cc1371ee1d8f9c33d922513be59b8a9115e48f3"}
Sep 09 16:18:23.353322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:23.353189    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:23.373382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:23.373338    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" podStartSLOduration=17.373321954 podStartE2EDuration="17.373321954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:23.372432714 +0000 UTC m=+283.322013560" watchObservedRunningTime="2026-09-09 16:18:23.373321954 +0000 UTC m=+283.322902800"
Sep 09 16:18:27.168696 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:27.168642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:27.169205 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:27.168740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:27.169205 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:27.168858    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:27.169205 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:27.168944    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls podName:44341d7e-e18f-416c-9eee-ecaa4010342b nodeName:}" failed. No retries permitted until 2026-09-09 16:18:43.16892405 +0000 UTC m=+303.118504888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-94ss4" (UID: "44341d7e-e18f-416c-9eee-ecaa4010342b") : secret "insights-runtime-extractor-tls" not found
Sep 09 16:18:27.169205 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:27.169184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44341d7e-e18f-416c-9eee-ecaa4010342b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:29.373111 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:29.373083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x9bq" event={"ID":"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5","Type":"ContainerStarted","Data":"e7ab7a80e32eaad683229bd29f3afd83059698d19f359f969fd36dad4df13b72"}
Sep 09 16:18:29.373432 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:29.373134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5x9bq" event={"ID":"cb2d6b57-09e3-438f-be2a-93fcc6aea9e5","Type":"ContainerStarted","Data":"4bc60f4c28ad85f751d05f704d8ae99a941e055b54a92499c4ab0ac55ba13566"}
Sep 09 16:18:29.373432 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:29.373204    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:18:29.388378 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:29.388335    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5x9bq" podStartSLOduration=251.926200022 podStartE2EDuration="4m18.388320336s" totalImagesPullingTime="6.462120314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:29.387267731 +0000 UTC m=+289.336848582" watchObservedRunningTime="2026-09-09 16:18:29.388320336 +0000 UTC m=+289.337901181"
Sep 09 16:18:33.742014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:33.741954    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:34.526945 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:34.526897    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mgf65" podUID="c6337926-86fe-44e6-82e4-2e7637dc1b70"
Sep 09 16:18:39.379859 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:39.379832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5x9bq"
Sep 09 16:18:40.459661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:40.459638    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:18:40.463292 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:40.463271    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:18:40.469021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:40.468998    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 16:18:43.171550 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.171517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:43.173927 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.173715    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44341d7e-e18f-416c-9eee-ecaa4010342b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-94ss4\" (UID: \"44341d7e-e18f-416c-9eee-ecaa4010342b\") " pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:43.199915 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.199891    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q2x6b\""
Sep 09 16:18:43.208110 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.208087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-94ss4"
Sep 09 16:18:43.474287 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.474215    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-94ss4"]
Sep 09 16:18:43.477142 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:43.477106    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44341d7e_e18f_416c_9eee_ecaa4010342b.slice/crio-bad93a15586e53a31bf33a61b36246d10ad622d7bfd7474d7f3b6ea374600608 WatchSource:0}: Error finding container bad93a15586e53a31bf33a61b36246d10ad622d7bfd7474d7f3b6ea374600608: Status 404 returned error can't find the container with id bad93a15586e53a31bf33a61b36246d10ad622d7bfd7474d7f3b6ea374600608
Sep 09 16:18:43.529388 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.529366    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:18:43.747310 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:43.747236    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:44.421010 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:44.420989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-94ss4" event={"ID":"44341d7e-e18f-416c-9eee-ecaa4010342b","Type":"ContainerStarted","Data":"d23e5cd5b9890ab0dddfc9e04faa9e2327b659614ccd60a3f27fb0610f77f92c"}
Sep 09 16:18:44.421281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:44.421039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-94ss4" event={"ID":"44341d7e-e18f-416c-9eee-ecaa4010342b","Type":"ContainerStarted","Data":"f748640c68e93056ef2ce40eeba07da52bdf831cae851ac557ecba30176183fa"}
Sep 09 16:18:44.421281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:44.421057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-94ss4" event={"ID":"44341d7e-e18f-416c-9eee-ecaa4010342b","Type":"ContainerStarted","Data":"bad93a15586e53a31bf33a61b36246d10ad622d7bfd7474d7f3b6ea374600608"}
Sep 09 16:18:46.428758 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:46.428664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-94ss4" event={"ID":"44341d7e-e18f-416c-9eee-ecaa4010342b","Type":"ContainerStarted","Data":"71782d2a5fef6aa5859837a5216899af761b3570454d39c6f360822f6137214b"}
Sep 09 16:18:46.444768 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:46.444721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-94ss4" podStartSLOduration=32.919325262 podStartE2EDuration="35.444704686s" totalImagesPullingTime="2.525379424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:18:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:46.444461264 +0000 UTC m=+306.394042111" watchObservedRunningTime="2026-09-09 16:18:46.444704686 +0000 UTC m=+306.394285532"
Sep 09 16:18:48.525226 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:48.525191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:18:52.644155 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.644121    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"]
Sep 09 16:18:52.647254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.647235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.650077 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650058    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 16:18:52.650907 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650871    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 16:18:52.650907 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ql66f\""
Sep 09 16:18:52.651070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650922    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 16:18:52.651070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 16:18:52.651070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.650988    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 16:18:52.658124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.658105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"]
Sep 09 16:18:52.742713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.742677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.742870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.742764    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v58c\" (UniqueName: \"kubernetes.io/projected/a7155a4c-fded-4a56-9b8f-797fd4c10444-kube-api-access-2v58c\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.742870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.742807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7155a4c-fded-4a56-9b8f-797fd4c10444-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.742870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.742854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.843191 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.843163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v58c\" (UniqueName: \"kubernetes.io/projected/a7155a4c-fded-4a56-9b8f-797fd4c10444-kube-api-access-2v58c\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.843191 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.843193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7155a4c-fded-4a56-9b8f-797fd4c10444-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.843352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.843215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.843352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.843319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.843886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.843866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7155a4c-fded-4a56-9b8f-797fd4c10444-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.845705 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.845681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.845814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.845736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7155a4c-fded-4a56-9b8f-797fd4c10444-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.853615 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.853595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v58c\" (UniqueName: \"kubernetes.io/projected/a7155a4c-fded-4a56-9b8f-797fd4c10444-kube-api-access-2v58c\") pod \"prometheus-operator-6f4946db56-rpk4m\" (UID: \"a7155a4c-fded-4a56-9b8f-797fd4c10444\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:52.956653 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:52.956583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"
Sep 09 16:18:53.223046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:53.222950    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rpk4m"]
Sep 09 16:18:53.227260 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:53.227236    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7155a4c_fded_4a56_9b8f_797fd4c10444.slice/crio-6a084f3c1e9c2af1207c1f203927c94cff9ce9366148e0f6c3bf561ab1b69452 WatchSource:0}: Error finding container 6a084f3c1e9c2af1207c1f203927c94cff9ce9366148e0f6c3bf561ab1b69452: Status 404 returned error can't find the container with id 6a084f3c1e9c2af1207c1f203927c94cff9ce9366148e0f6c3bf561ab1b69452
Sep 09 16:18:53.451246 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:53.451211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m" event={"ID":"a7155a4c-fded-4a56-9b8f-797fd4c10444","Type":"ContainerStarted","Data":"6a084f3c1e9c2af1207c1f203927c94cff9ce9366148e0f6c3bf561ab1b69452"}
Sep 09 16:18:57.464678 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:57.464646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m" event={"ID":"a7155a4c-fded-4a56-9b8f-797fd4c10444","Type":"ContainerStarted","Data":"772a1e1ff2e74d6ab9c0f91bee04246a161f09d0e3890dff0d761c47bd69801c"}
Sep 09 16:18:57.465092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:57.464697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m" event={"ID":"a7155a4c-fded-4a56-9b8f-797fd4c10444","Type":"ContainerStarted","Data":"33390189b4efe36eac278122a5d6d1ea4aa3e07fd187ef25e5aabf8e5c032b70"}
Sep 09 16:18:57.480144 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:57.480099    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-rpk4m" podStartSLOduration=1.8375177470000001 podStartE2EDuration="5.480083277s" totalImagesPullingTime="3.64256553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:18:57.479376475 +0000 UTC m=+317.428957321" watchObservedRunningTime="2026-09-09 16:18:57.480083277 +0000 UTC m=+317.429664122"
Sep 09 16:18:58.760456 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:58.760404    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" podUID="35d36563-9837-400b-bc6c-fa67b73c66f6" containerName="registry" containerID="cri-o://fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98" gracePeriod=30
Sep 09 16:18:59.001525 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.001504    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:59.040555 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.040525    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-27wdt"]
Sep 09 16:18:59.040867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.040850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d36563-9837-400b-bc6c-fa67b73c66f6" containerName="registry"
Sep 09 16:18:59.040939 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.040871    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d36563-9837-400b-bc6c-fa67b73c66f6" containerName="registry"
Sep 09 16:18:59.040939 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.040933    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d36563-9837-400b-bc6c-fa67b73c66f6" containerName="registry"
Sep 09 16:18:59.044166 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.044138    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"]
Sep 09 16:18:59.044299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.044281    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.046601 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.046583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zlvb9\""
Sep 09 16:18:59.046719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.046636    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 16:18:59.046719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.046711    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 16:18:59.046839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.046720    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 16:18:59.047282 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.047266    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.049605 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.049588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 09 16:18:59.049829 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.049774    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 09 16:18:59.049829 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.049774    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s7qnw\""
Sep 09 16:18:59.050017 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.049893    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 09 16:18:59.055365 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.055347    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"]
Sep 09 16:18:59.083364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083343    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj7xc\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083411    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083630 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083630 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083500    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083630 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083630 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083567    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token\") pod \"35d36563-9837-400b-bc6c-fa67b73c66f6\" (UID: \"35d36563-9837-400b-bc6c-fa67b73c66f6\") "
Sep 09 16:18:59.083855 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083828    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:18:59.084101 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.083879    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:18:59.086568 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.086542    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:18:59.086677 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.086658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:18:59.086742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.086658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:18:59.086799 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.086758    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc" (OuterVolumeSpecName: "kube-api-access-bj7xc") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "kube-api-access-bj7xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:18:59.086799 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.086760    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:18:59.092571 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.092542    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "35d36563-9837-400b-bc6c-fa67b73c66f6" (UID: "35d36563-9837-400b-bc6c-fa67b73c66f6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:18:59.184517 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxd6\" (UniqueName: \"kubernetes.io/projected/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-api-access-cqxd6\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-root\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-wtmp\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.184684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.184861 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184737    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-sys\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184861 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-textfile\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184861 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.184861 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.184861 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww745\" (UniqueName: \"kubernetes.io/projected/7009d4af-e4c2-491a-8135-ae30ff707f06-kube-api-access-ww745\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-metrics-client-ca\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-tls\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184970    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-installation-pull-secrets\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184980    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-bound-sa-token\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184988    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/35d36563-9837-400b-bc6c-fa67b73c66f6-ca-trust-extracted\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.184996    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7xc\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-kube-api-access-bj7xc\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.185014    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-trusted-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.185025    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-certificates\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.185035    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/35d36563-9837-400b-bc6c-fa67b73c66f6-image-registry-private-configuration\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.185299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.185045    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/35d36563-9837-400b-bc6c-fa67b73c66f6-registry-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:18:59.285500 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-sys\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285500 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-textfile\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285553    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww745\" (UniqueName: \"kubernetes.io/projected/7009d4af-e4c2-491a-8135-ae30ff707f06-kube-api-access-ww745\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285569    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-metrics-client-ca\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285591    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-sys\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-tls\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.285683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxd6\" (UniqueName: \"kubernetes.io/projected/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-api-access-cqxd6\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-root\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-wtmp\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-textfile\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.285917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-root\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286085    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.286459 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286274    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-wtmp\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286459 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-metrics-client-ca\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286604 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.286641 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286598    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-accelerators-collector-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.286911 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.286886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.288114 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.288093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-tls\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.288202 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.288185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.288256 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.288228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.288460 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.288441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7009d4af-e4c2-491a-8135-ae30ff707f06-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.292711 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.292670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxd6\" (UniqueName: \"kubernetes.io/projected/a60e7d91-7675-4ef9-b4f4-1b0aae7517e8-kube-api-access-cqxd6\") pod \"kube-state-metrics-74cb57df7-fcsgn\" (UID: \"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.292777 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.292761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww745\" (UniqueName: \"kubernetes.io/projected/7009d4af-e4c2-491a-8135-ae30ff707f06-kube-api-access-ww745\") pod \"node-exporter-27wdt\" (UID: \"7009d4af-e4c2-491a-8135-ae30ff707f06\") " pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.354656 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.354637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-27wdt"
Sep 09 16:18:59.359222 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.359101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"
Sep 09 16:18:59.366427 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:59.366407    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7009d4af_e4c2_491a_8135_ae30ff707f06.slice/crio-7d93bb5264ffae20aa6c18fa146c41dd442e58865b58dad79f027c0640a81e27 WatchSource:0}: Error finding container 7d93bb5264ffae20aa6c18fa146c41dd442e58865b58dad79f027c0640a81e27: Status 404 returned error can't find the container with id 7d93bb5264ffae20aa6c18fa146c41dd442e58865b58dad79f027c0640a81e27
Sep 09 16:18:59.472298 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.472258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27wdt" event={"ID":"7009d4af-e4c2-491a-8135-ae30ff707f06","Type":"ContainerStarted","Data":"7d93bb5264ffae20aa6c18fa146c41dd442e58865b58dad79f027c0640a81e27"}
Sep 09 16:18:59.473374 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.473349    2559 generic.go:356] "Generic (PLEG): container finished" podID="35d36563-9837-400b-bc6c-fa67b73c66f6" containerID="fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98" exitCode=0
Sep 09 16:18:59.473479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.473395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" event={"ID":"35d36563-9837-400b-bc6c-fa67b73c66f6","Type":"ContainerDied","Data":"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"}
Sep 09 16:18:59.473479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.473399    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf"
Sep 09 16:18:59.473479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.473435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c9c9565fd-nzxmf" event={"ID":"35d36563-9837-400b-bc6c-fa67b73c66f6","Type":"ContainerDied","Data":"947851d46c1571926fb463186cc1371ee1d8f9c33d922513be59b8a9115e48f3"}
Sep 09 16:18:59.473479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.473473    2559 scope.go:117] "RemoveContainer" containerID="fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"
Sep 09 16:18:59.480874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.480855    2559 scope.go:117] "RemoveContainer" containerID="fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"
Sep 09 16:18:59.481135 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:18:59.481113    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98\": container with ID starting with fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98 not found: ID does not exist" containerID="fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"
Sep 09 16:18:59.481199 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.481143    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98"} err="failed to get container status \"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98\": rpc error: code = NotFound desc = could not find container \"fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98\": container with ID starting with fa616e516a2a0d52252bec2a49beb29364dd44378eb00da4190e16d79284db98 not found: ID does not exist"
Sep 09 16:18:59.492400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.492380    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:59.496572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.496550    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c9c9565fd-nzxmf"]
Sep 09 16:18:59.626735 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:18:59.626678    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn"]
Sep 09 16:18:59.629348 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:18:59.629330    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e7d91_7675_4ef9_b4f4_1b0aae7517e8.slice/crio-719503834bb8bb5ca9ba5da64c0092a53df481b456c476de37e2235341c5d9a3 WatchSource:0}: Error finding container 719503834bb8bb5ca9ba5da64c0092a53df481b456c476de37e2235341c5d9a3: Status 404 returned error can't find the container with id 719503834bb8bb5ca9ba5da64c0092a53df481b456c476de37e2235341c5d9a3
Sep 09 16:19:00.479117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.479081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn" event={"ID":"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8","Type":"ContainerStarted","Data":"719503834bb8bb5ca9ba5da64c0092a53df481b456c476de37e2235341c5d9a3"}
Sep 09 16:19:00.530725 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.530692    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d36563-9837-400b-bc6c-fa67b73c66f6" path="/var/lib/kubelet/pods/35d36563-9837-400b-bc6c-fa67b73c66f6/volumes"
Sep 09 16:19:00.988989 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.988943    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-66875b9776-drjwh"]
Sep 09 16:19:00.992592 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.992574    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:00.995322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995289    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 16:19:00.995322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995318    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-579q4o36ia0o0\""
Sep 09 16:19:00.995505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995289    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 16:19:00.995649 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995631    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7wf7z\""
Sep 09 16:19:00.995773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 16:19:00.995773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.995768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 16:19:00.996384 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:00.996366    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 16:19:01.003576 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.003555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66875b9776-drjwh"]
Sep 09 16:19:01.107006 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.106981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-metrics-client-ca\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqsbv\" (UniqueName: \"kubernetes.io/projected/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-kube-api-access-xqsbv\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-grpc-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107274 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107174    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.107274 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.107193    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.207619 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.207584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.207742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.207632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.207742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.207662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-grpc-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.207742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.207696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.207742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.207720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.208335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.208161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.208335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.208200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-metrics-client-ca\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.208335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.208252    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqsbv\" (UniqueName: \"kubernetes.io/projected/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-kube-api-access-xqsbv\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.209203 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.209174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-metrics-client-ca\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.211283 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.210692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.211393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.211350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.211697 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.211665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.212001 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.211976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.212839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.212801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-grpc-tls\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.213602 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.213560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.217848 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.217825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqsbv\" (UniqueName: \"kubernetes.io/projected/a9b9f82f-6815-4a95-89dd-c8a666bc5f35-kube-api-access-xqsbv\") pod \"thanos-querier-66875b9776-drjwh\" (UID: \"a9b9f82f-6815-4a95-89dd-c8a666bc5f35\") " pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.303019 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.302931    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:01.485252 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.485221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn" event={"ID":"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8","Type":"ContainerStarted","Data":"277ee787f45b714d522d54f7618d63c39bee5d38e20fdf31de77d545ee5aee90"}
Sep 09 16:19:01.485663 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.485324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn" event={"ID":"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8","Type":"ContainerStarted","Data":"a33f94a6300a243923882aa063eb68c2fb3d324afc8e9576bcd2fe1d5b5ae852"}
Sep 09 16:19:01.485663 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.485352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn" event={"ID":"a60e7d91-7675-4ef9-b4f4-1b0aae7517e8","Type":"ContainerStarted","Data":"2834ecf4bdcee468bcb0f12f1e7903ee2190f000df7ed0db2acb22e3bcb41e82"}
Sep 09 16:19:01.486883 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.486853    2559 generic.go:356] "Generic (PLEG): container finished" podID="7009d4af-e4c2-491a-8135-ae30ff707f06" containerID="4e4289f0db4b90b37b871e15caa7a85ee0d78579680027a108df6f08f9159706" exitCode=0
Sep 09 16:19:01.487024 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.486890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27wdt" event={"ID":"7009d4af-e4c2-491a-8135-ae30ff707f06","Type":"ContainerDied","Data":"4e4289f0db4b90b37b871e15caa7a85ee0d78579680027a108df6f08f9159706"}
Sep 09 16:19:01.505108 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.505057    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fcsgn" podStartSLOduration=1.05277787 podStartE2EDuration="2.50504101s" totalImagesPullingTime="1.45226314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:01.50447521 +0000 UTC m=+321.454056057" watchObservedRunningTime="2026-09-09 16:19:01.50504101 +0000 UTC m=+321.454621856"
Sep 09 16:19:01.576871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:01.576847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66875b9776-drjwh"]
Sep 09 16:19:01.580010 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:19:01.579988    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b9f82f_6815_4a95_89dd_c8a666bc5f35.slice/crio-411460ca785132ff627543c58df7b86a3b0ebadc7ea897e0acd65d59b7fb0912 WatchSource:0}: Error finding container 411460ca785132ff627543c58df7b86a3b0ebadc7ea897e0acd65d59b7fb0912: Status 404 returned error can't find the container with id 411460ca785132ff627543c58df7b86a3b0ebadc7ea897e0acd65d59b7fb0912
Sep 09 16:19:02.493384 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:02.493345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27wdt" event={"ID":"7009d4af-e4c2-491a-8135-ae30ff707f06","Type":"ContainerStarted","Data":"edd72659eb21db0cdfe4c9c96f8a56629b6c27410443d7da914ff86d687ed13c"}
Sep 09 16:19:02.493818 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:02.493411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-27wdt" event={"ID":"7009d4af-e4c2-491a-8135-ae30ff707f06","Type":"ContainerStarted","Data":"5a01ad3875f8a290f83fabbf21662a46f2e4cecc8f27e52791e40ca035efea12"}
Sep 09 16:19:02.494718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:02.494692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"411460ca785132ff627543c58df7b86a3b0ebadc7ea897e0acd65d59b7fb0912"}
Sep 09 16:19:02.513028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:02.512976    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-27wdt" podStartSLOduration=2.174402528 podStartE2EDuration="3.512948335s" totalImagesPullingTime="1.2689971s" totalInitContainerRuntime="69.548707ms" isStatefulPod=true podCreationTimestamp="2026-09-09 16:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:02.511045889 +0000 UTC m=+322.460626737" watchObservedRunningTime="2026-09-09 16:19:02.512948335 +0000 UTC m=+322.462529179"
Sep 09 16:19:03.504371 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.504344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"20f1aaf7cc149195a7247c5393e441ee9fdfbd6816e1494c4451c5770b07824f"}
Sep 09 16:19:03.504641 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.504430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"6586b18d55d17dcf135c0b9895255be6086777f962a1ac1444a09af73d74afb9"}
Sep 09 16:19:03.770120 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.770042    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"]
Sep 09 16:19:03.772788 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.772773    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:03.775434 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.775412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 16:19:03.775536 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.775439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nhrrt\""
Sep 09 16:19:03.781688 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.781667    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"]
Sep 09 16:19:03.930919 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.930884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/462130c6-8e4c-4397-8d04-b317f1cbcc7a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:03.931121 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:03.930980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhqd\" (UniqueName: \"kubernetes.io/projected/462130c6-8e4c-4397-8d04-b317f1cbcc7a-kube-api-access-mmhqd\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.031640 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.031621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhqd\" (UniqueName: \"kubernetes.io/projected/462130c6-8e4c-4397-8d04-b317f1cbcc7a-kube-api-access-mmhqd\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.031760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.031670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/462130c6-8e4c-4397-8d04-b317f1cbcc7a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.033890 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.033873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/462130c6-8e4c-4397-8d04-b317f1cbcc7a-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.040171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.040149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhqd\" (UniqueName: \"kubernetes.io/projected/462130c6-8e4c-4397-8d04-b317f1cbcc7a-kube-api-access-mmhqd\") pod \"monitoring-plugin-85bbbbbf4-nfs4j\" (UID: \"462130c6-8e4c-4397-8d04-b317f1cbcc7a\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.081364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.081345    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:04.348396 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.348323    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"]
Sep 09 16:19:04.350767 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:19:04.350743    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462130c6_8e4c_4397_8d04_b317f1cbcc7a.slice/crio-243e4595194f3e55d9b20eb3519ee3795b37462fb713d9e0b50c02f111b98a00 WatchSource:0}: Error finding container 243e4595194f3e55d9b20eb3519ee3795b37462fb713d9e0b50c02f111b98a00: Status 404 returned error can't find the container with id 243e4595194f3e55d9b20eb3519ee3795b37462fb713d9e0b50c02f111b98a00
Sep 09 16:19:04.511278 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.511245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"a96a2895fde01cd8568c6a050fec5f3d6bc0992ac7f17b7ffa4cb704105e1c1c"}
Sep 09 16:19:04.512563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:04.512536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j" event={"ID":"462130c6-8e4c-4397-8d04-b317f1cbcc7a","Type":"ContainerStarted","Data":"243e4595194f3e55d9b20eb3519ee3795b37462fb713d9e0b50c02f111b98a00"}
Sep 09 16:19:05.522603 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:05.521831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"04fb3413cfc9b5bc66e57a0923c5971c73f075c40699b16a30ca25069416178f"}
Sep 09 16:19:05.522603 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:05.521909    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"09f0a40f5120170699f39e24732e59cb3093b795d2287d1786d46f392cd01751"}
Sep 09 16:19:05.523105 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:05.521949    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" event={"ID":"a9b9f82f-6815-4a95-89dd-c8a666bc5f35","Type":"ContainerStarted","Data":"6a2b941fb688e794bf76935562760ff2db2406b5e4fa48001935961ac7607504"}
Sep 09 16:19:05.523105 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:05.522873    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:05.544337 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:05.544281    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh" podStartSLOduration=2.570172234 podStartE2EDuration="5.544259856s" totalImagesPullingTime="2.974087622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:05.543856707 +0000 UTC m=+325.493437567" watchObservedRunningTime="2026-09-09 16:19:05.544259856 +0000 UTC m=+325.493840729"
Sep 09 16:19:06.531240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:06.531211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j" event={"ID":"462130c6-8e4c-4397-8d04-b317f1cbcc7a","Type":"ContainerStarted","Data":"7d9c5bd6ca9fde1d3f979c2b9eee1f6ad58bb20b5f29bb42e0a897d688a5ba37"}
Sep 09 16:19:06.544305 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:06.544263    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j" podStartSLOduration=2.081098658 podStartE2EDuration="3.544247686s" totalImagesPullingTime="1.463149028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:06.543652711 +0000 UTC m=+326.493233557" watchObservedRunningTime="2026-09-09 16:19:06.544247686 +0000 UTC m=+326.493828574"
Sep 09 16:19:07.532032 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:07.532001    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:07.536336 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:07.536315    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-nfs4j"
Sep 09 16:19:12.537599 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:12.537568    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-66875b9776-drjwh"
Sep 09 16:19:23.750686 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:23.750640    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:19:33.751221 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:33.751141    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:19:43.751119 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:43.751076    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 16:19:43.751585 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:43.751158    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z"
Sep 09 16:19:43.751633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:43.751597    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6f67d12fb5bef90d0fe1a44b0002697f70cc1a183877adece47ed39155af8a70"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 16:19:43.751667 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:43.751629    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" podUID="2712911f-e0fb-49ea-b1bb-a7f59e073645" containerName="service-proxy" containerID="cri-o://6f67d12fb5bef90d0fe1a44b0002697f70cc1a183877adece47ed39155af8a70" gracePeriod=30
Sep 09 16:19:44.655041 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:44.655002    2559 generic.go:356] "Generic (PLEG): container finished" podID="2712911f-e0fb-49ea-b1bb-a7f59e073645" containerID="6f67d12fb5bef90d0fe1a44b0002697f70cc1a183877adece47ed39155af8a70" exitCode=2
Sep 09 16:19:44.655208 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:44.655056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerDied","Data":"6f67d12fb5bef90d0fe1a44b0002697f70cc1a183877adece47ed39155af8a70"}
Sep 09 16:19:44.655208 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:44.655118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86476f7b4f-5nm8z" event={"ID":"2712911f-e0fb-49ea-b1bb-a7f59e073645","Type":"ContainerStarted","Data":"b68fc2e2fa0cd1a1ac43db515d7ba7765a05bdf36aba2040febda543051e659e"}
Sep 09 16:19:53.286886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.286846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:19:53.289135 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.289108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6337926-86fe-44e6-82e4-2e7637dc1b70-metrics-certs\") pod \"network-metrics-daemon-mgf65\" (UID: \"c6337926-86fe-44e6-82e4-2e7637dc1b70\") " pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:19:53.328798 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.328766    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cjzmn\""
Sep 09 16:19:53.336772 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.336751    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgf65"
Sep 09 16:19:53.641682 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.641614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgf65"]
Sep 09 16:19:53.644891 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:19:53.644861    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6337926_86fe_44e6_82e4_2e7637dc1b70.slice/crio-a3cba2b64df64c26fd97d19f877bb0f404b19ffebdf91c64b506c4c4e6dcfa74 WatchSource:0}: Error finding container a3cba2b64df64c26fd97d19f877bb0f404b19ffebdf91c64b506c4c4e6dcfa74: Status 404 returned error can't find the container with id a3cba2b64df64c26fd97d19f877bb0f404b19ffebdf91c64b506c4c4e6dcfa74
Sep 09 16:19:53.685196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:53.685162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgf65" event={"ID":"c6337926-86fe-44e6-82e4-2e7637dc1b70","Type":"ContainerStarted","Data":"a3cba2b64df64c26fd97d19f877bb0f404b19ffebdf91c64b506c4c4e6dcfa74"}
Sep 09 16:19:57.703847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:57.703811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgf65" event={"ID":"c6337926-86fe-44e6-82e4-2e7637dc1b70","Type":"ContainerStarted","Data":"15af92b376cc59f489b555e0ee31a7fed598f708110b5513dd96db41466e7874"}
Sep 09 16:19:57.704245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:57.703865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgf65" event={"ID":"c6337926-86fe-44e6-82e4-2e7637dc1b70","Type":"ContainerStarted","Data":"078cf525d9177b5154ebc0da894569d23a5b00f72eb050f6f93f77055c3cfa75"}
Sep 09 16:19:57.719866 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:19:57.719821    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mgf65" podStartSLOduration=374.328042076 podStartE2EDuration="6m17.719806133s" totalImagesPullingTime="3.391764057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:19:57.718479156 +0000 UTC m=+377.668060001" watchObservedRunningTime="2026-09-09 16:19:57.719806133 +0000 UTC m=+377.669387021"
Sep 09 16:20:21.548155 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.548122    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 16:20:21.551731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.551709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.553951 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.553926    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 16:20:21.554350 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.554301    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 16:20:21.554449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.554398    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 16:20:21.554798 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.554777    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 16:20:21.554907 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.554809    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 16:20:21.554998 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.554981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dsfnj\""
Sep 09 16:20:21.555079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.555005    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 16:20:21.555079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.555039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 16:20:21.555244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.555225    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 16:20:21.558697 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.558679    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 16:20:21.564737 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.564718    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 16:20:21.696548 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-config-volume\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696548 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696660    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-web-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.696805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.697123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-config-out\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.697123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.697123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rq9l\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-kube-api-access-2rq9l\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.697123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.696948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-config-volume\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797639    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.797814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-web-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797833    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-config-out\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rq9l\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-kube-api-access-2rq9l\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798082 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.797999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798388 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.798137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798854 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.798825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.798985 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.798902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a33105c4-d146-423f-b135-67430ac6c250-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.800587 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.800561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.800923 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.800896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a33105c4-d146-423f-b135-67430ac6c250-config-out\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801101 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801266 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801496 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-config-volume\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801496 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801743 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-web-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.801810 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.801765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.802578 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.802556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a33105c4-d146-423f-b135-67430ac6c250-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.806403 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.806381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rq9l\" (UniqueName: \"kubernetes.io/projected/a33105c4-d146-423f-b135-67430ac6c250-kube-api-access-2rq9l\") pod \"alertmanager-main-0\" (UID: \"a33105c4-d146-423f-b135-67430ac6c250\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:21.862271 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:21.862248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 16:20:22.140912 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:22.140884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 16:20:22.142110 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:20:22.142085    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33105c4_d146_423f_b135_67430ac6c250.slice/crio-489327930d3830d067783f8c5906bdffdf7fafc06f73183bda5457f8f633783d WatchSource:0}: Error finding container 489327930d3830d067783f8c5906bdffdf7fafc06f73183bda5457f8f633783d: Status 404 returned error can't find the container with id 489327930d3830d067783f8c5906bdffdf7fafc06f73183bda5457f8f633783d
Sep 09 16:20:22.791992 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:22.791943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"489327930d3830d067783f8c5906bdffdf7fafc06f73183bda5457f8f633783d"}
Sep 09 16:20:23.797196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:23.797158    2559 generic.go:356] "Generic (PLEG): container finished" podID="a33105c4-d146-423f-b135-67430ac6c250" containerID="3b6c91f38f7f69e07dfd03aa1d7bca7cea389c39443d65e199b5d6a2a3176065" exitCode=0
Sep 09 16:20:23.797647 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:23.797230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerDied","Data":"3b6c91f38f7f69e07dfd03aa1d7bca7cea389c39443d65e199b5d6a2a3176065"}
Sep 09 16:20:28.816755 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:28.816727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"a9256582c7b09941c38f16741517f268ea22c6b2b944ac3ea13f37caaef2cebd"}
Sep 09 16:20:29.823436 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.823403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"d8fc0dfee3d14cd6ab2213b1c271372a72ae59af6ee53df8c618d7440131e3c6"}
Sep 09 16:20:29.823806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.823491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"69afdcbf3d0d06ebfbad37cca3bf1e75b14fde4589be7a7a646b832f5544ede0"}
Sep 09 16:20:29.823806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.823517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"1ca19c9e2a4082a933b4d1d79a83b83d1ea93f79f5cc989dd78b76c04f3e1d5b"}
Sep 09 16:20:29.823806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.823545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"f9ca718fe5347566ef838b5e0e623795ea4148a607a8a879fd9f2d41286c3a6e"}
Sep 09 16:20:29.823806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.823570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a33105c4-d146-423f-b135-67430ac6c250","Type":"ContainerStarted","Data":"097cd72c7fe664100a68bad1d52f00689b6145d3bb1d61cf6f4054a47823eb3b"}
Sep 09 16:20:29.849200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:20:29.849152    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.555217957 podStartE2EDuration="8.849135756s" totalImagesPullingTime="6.27310071s" totalInitContainerRuntime="20.817089ms" isStatefulPod=false podCreationTimestamp="2026-09-09 16:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:20:29.847286389 +0000 UTC m=+409.796867237" watchObservedRunningTime="2026-09-09 16:20:29.849135756 +0000 UTC m=+409.798716601"
Sep 09 16:23:37.726051 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.726018    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bxrh5"]
Sep 09 16:23:37.729075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.729050    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.731796 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.731773    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 16:23:37.736397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.736367    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t8vh8\""
Sep 09 16:23:37.736516 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.736451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 16:23:37.738626 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.738602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bxrh5"]
Sep 09 16:23:37.753245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.753222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.753333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.753261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjg9d\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-kube-api-access-rjg9d\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.853687 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.853654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.853821 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.853697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjg9d\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-kube-api-access-rjg9d\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.868456 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.868429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:37.868560 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:37.868489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjg9d\" (UniqueName: \"kubernetes.io/projected/2e16b443-17d8-4f18-83a2-430a1d64a9b8-kube-api-access-rjg9d\") pod \"cert-manager-webhook-6486ffd997-bxrh5\" (UID: \"2e16b443-17d8-4f18-83a2-430a1d64a9b8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:38.042498 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:38.042472    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:38.312765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:38.312687    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bxrh5"]
Sep 09 16:23:38.315536 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:23:38.315509    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e16b443_17d8_4f18_83a2_430a1d64a9b8.slice/crio-8c7868b2b0981c2e62fbcb71cf20c42c158757d46830cd9a54c62de49e2051ba WatchSource:0}: Error finding container 8c7868b2b0981c2e62fbcb71cf20c42c158757d46830cd9a54c62de49e2051ba: Status 404 returned error can't find the container with id 8c7868b2b0981c2e62fbcb71cf20c42c158757d46830cd9a54c62de49e2051ba
Sep 09 16:23:38.484779 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:38.484745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5" event={"ID":"2e16b443-17d8-4f18-83a2-430a1d64a9b8","Type":"ContainerStarted","Data":"8c7868b2b0981c2e62fbcb71cf20c42c158757d46830cd9a54c62de49e2051ba"}
Sep 09 16:23:41.347916 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:41.347888    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:23:41.348508 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:41.348489    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:23:41.500583 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:41.500549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5" event={"ID":"2e16b443-17d8-4f18-83a2-430a1d64a9b8","Type":"ContainerStarted","Data":"003e5aee7bbd04f7add042ddbca6d498f59d67367dcf2761644ca2c9a1498e44"}
Sep 09 16:23:41.500730 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:41.500668    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:23:41.517344 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:41.517300    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5" podStartSLOduration=1.442269879 podStartE2EDuration="4.517285129s" totalImagesPullingTime="3.07501525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:23:41.516804514 +0000 UTC m=+601.466385395" watchObservedRunningTime="2026-09-09 16:23:41.517285129 +0000 UTC m=+601.466865976"
Sep 09 16:23:47.508725 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:23:47.508692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bxrh5"
Sep 09 16:24:09.235686 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.235587    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"]
Sep 09 16:24:09.238649 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.238634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.243205 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.243178    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 16:24:09.243388 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.243372    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 16:24:09.244066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.244050    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 16:24:09.245294 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.245273    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 16:24:09.245703 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.245685    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 16:24:09.246659 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.246644    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mlmxp\""
Sep 09 16:24:09.261672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.261651    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"]
Sep 09 16:24:09.297935 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.297911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.298048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.297946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-metrics-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.298048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.297993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2lj\" (UniqueName: \"kubernetes.io/projected/aab139f6-c5b9-443f-83a0-9a8760b10173-kube-api-access-2f2lj\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.298117 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.298053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aab139f6-c5b9-443f-83a0-9a8760b10173-manager-config\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.398726 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.398698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aab139f6-c5b9-443f-83a0-9a8760b10173-manager-config\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.398845 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.398748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.398915 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.398860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-metrics-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.398915 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.398894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2lj\" (UniqueName: \"kubernetes.io/projected/aab139f6-c5b9-443f-83a0-9a8760b10173-kube-api-access-2f2lj\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.399465 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.399445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/aab139f6-c5b9-443f-83a0-9a8760b10173-manager-config\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.401195 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.401174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-metrics-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.401285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.401271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aab139f6-c5b9-443f-83a0-9a8760b10173-cert\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.408230 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.408211    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2lj\" (UniqueName: \"kubernetes.io/projected/aab139f6-c5b9-443f-83a0-9a8760b10173-kube-api-access-2f2lj\") pod \"lws-controller-manager-6bb9f9f6c8-fsh5j\" (UID: \"aab139f6-c5b9-443f-83a0-9a8760b10173\") " pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.547731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.547701    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:09.815390 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.815272    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"]
Sep 09 16:24:09.817894 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:24:09.817867    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab139f6_c5b9_443f_83a0_9a8760b10173.slice/crio-8b51e7b7b4c74bceb3d424bd0a14955ae5edbf2e7b0153876d1ccc5698cfb9be WatchSource:0}: Error finding container 8b51e7b7b4c74bceb3d424bd0a14955ae5edbf2e7b0153876d1ccc5698cfb9be: Status 404 returned error can't find the container with id 8b51e7b7b4c74bceb3d424bd0a14955ae5edbf2e7b0153876d1ccc5698cfb9be
Sep 09 16:24:09.819756 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:09.819741    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:24:10.328226 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.328190    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"]
Sep 09 16:24:10.332439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.332421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.338975 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.338842    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 16:24:10.339063 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.338999    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 16:24:10.339122 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.339087    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 16:24:10.339122 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.339087    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q77nk\""
Sep 09 16:24:10.339214 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.339092    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 16:24:10.346232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.346209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"]
Sep 09 16:24:10.406977 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.406935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.407139 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.407014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng88\" (UniqueName: \"kubernetes.io/projected/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-kube-api-access-wng88\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.407139 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.407106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.407251 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.407180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.407251 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.407209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508108 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng88\" (UniqueName: \"kubernetes.io/projected/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-kube-api-access-wng88\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.508640 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.508617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-manifests\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.510499 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.510472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-tmp\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.510584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.510497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.510584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.510546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.515897 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.515878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng88\" (UniqueName: \"kubernetes.io/projected/4e72ed5a-0443-4c43-bbaa-005584ec8cfc-kube-api-access-wng88\") pod \"opendatahub-operator-controller-manager-b9b77fc68-gsv4t\" (UID: \"4e72ed5a-0443-4c43-bbaa-005584ec8cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.613448 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.613371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j" event={"ID":"aab139f6-c5b9-443f-83a0-9a8760b10173","Type":"ContainerStarted","Data":"8b51e7b7b4c74bceb3d424bd0a14955ae5edbf2e7b0153876d1ccc5698cfb9be"}
Sep 09 16:24:10.643037 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.643010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:10.924836 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:10.924787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"]
Sep 09 16:24:10.927866 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:24:10.927833    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e72ed5a_0443_4c43_bbaa_005584ec8cfc.slice/crio-9b6f175f058a00feefd7c87900df341a304316452451624aadf817531bf572e6 WatchSource:0}: Error finding container 9b6f175f058a00feefd7c87900df341a304316452451624aadf817531bf572e6: Status 404 returned error can't find the container with id 9b6f175f058a00feefd7c87900df341a304316452451624aadf817531bf572e6
Sep 09 16:24:11.619323 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:11.619281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t" event={"ID":"4e72ed5a-0443-4c43-bbaa-005584ec8cfc","Type":"ContainerStarted","Data":"9b6f175f058a00feefd7c87900df341a304316452451624aadf817531bf572e6"}
Sep 09 16:24:12.629699 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:12.629595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j" event={"ID":"aab139f6-c5b9-443f-83a0-9a8760b10173","Type":"ContainerStarted","Data":"6c4595e8e2a38af0386cb4b8369cfe159c7d56e957cb8041d7467392b5781e67"}
Sep 09 16:24:12.630122 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:12.629766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:12.647157 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:12.647089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j" podStartSLOduration=1.107844675 podStartE2EDuration="3.647071662s" totalImagesPullingTime="2.539226987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:12.645976301 +0000 UTC m=+632.595557156" watchObservedRunningTime="2026-09-09 16:24:12.647071662 +0000 UTC m=+632.596652512"
Sep 09 16:24:15.649836 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:15.649791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t" event={"ID":"4e72ed5a-0443-4c43-bbaa-005584ec8cfc","Type":"ContainerStarted","Data":"25fd8ddcdd24a87360508d1eaca0f038d5e14b55863c38306f713756e6439f94"}
Sep 09 16:24:16.655548 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:16.655456    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e72ed5a-0443-4c43-bbaa-005584ec8cfc" containerID="25fd8ddcdd24a87360508d1eaca0f038d5e14b55863c38306f713756e6439f94" exitCode=0
Sep 09 16:24:16.655908 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:16.655548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t" event={"ID":"4e72ed5a-0443-4c43-bbaa-005584ec8cfc","Type":"ContainerDied","Data":"25fd8ddcdd24a87360508d1eaca0f038d5e14b55863c38306f713756e6439f94"}
Sep 09 16:24:17.661173 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:17.661138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t" event={"ID":"4e72ed5a-0443-4c43-bbaa-005584ec8cfc","Type":"ContainerStarted","Data":"059c859cc128d8ddb230d397522a3b5b11bd0f90b4848d3c78086f53ce3fbded"}
Sep 09 16:24:17.661570 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:17.661260    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:17.681706 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:17.681651    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t" podStartSLOduration=1.970589952 podStartE2EDuration="7.681630133s" totalImagesPullingTime="4.63665498s" totalInitContainerRuntime="1.074385201s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:17.680028549 +0000 UTC m=+637.629609396" watchObservedRunningTime="2026-09-09 16:24:17.681630133 +0000 UTC m=+637.631210980"
Sep 09 16:24:23.637338 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:23.637309    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6bb9f9f6c8-fsh5j"
Sep 09 16:24:28.667624 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:28.667590    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b9b77fc68-gsv4t"
Sep 09 16:24:31.609552 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.609520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"]
Sep 09 16:24:31.618151 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.618124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.621580 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.621556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 16:24:31.621692 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.621617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 16:24:31.622621 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.622602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 16:24:31.622732 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.622602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mcnhq\""
Sep 09 16:24:31.622732 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.622636    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 16:24:31.630917 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.630896    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"]
Sep 09 16:24:31.688090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.688049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tls-certs\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.688090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.688092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbzf\" (UniqueName: \"kubernetes.io/projected/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-kube-api-access-fpbzf\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.688309 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.688162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tmp\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.788706 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.788671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tmp\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.788874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.788729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tls-certs\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.788874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.788752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbzf\" (UniqueName: \"kubernetes.io/projected/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-kube-api-access-fpbzf\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.790954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.790934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tmp\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.791275 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.791243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-tls-certs\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.796626 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.796602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbzf\" (UniqueName: \"kubernetes.io/projected/eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50-kube-api-access-fpbzf\") pod \"kube-auth-proxy-769d969b5b-w77ms\" (UID: \"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50\") " pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:31.927715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:31.927626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"
Sep 09 16:24:32.201802 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:32.201731    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769d969b5b-w77ms"]
Sep 09 16:24:32.204969 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:24:32.204931    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa0f8c7_dc07_4d54_8df5_b1ea2e06eb50.slice/crio-30ab423b43b7a664b1de7f28098e28c6c4dd7460ab7d1887fe42a301368a0f24 WatchSource:0}: Error finding container 30ab423b43b7a664b1de7f28098e28c6c4dd7460ab7d1887fe42a301368a0f24: Status 404 returned error can't find the container with id 30ab423b43b7a664b1de7f28098e28c6c4dd7460ab7d1887fe42a301368a0f24
Sep 09 16:24:32.720215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:32.720173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms" event={"ID":"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50","Type":"ContainerStarted","Data":"30ab423b43b7a664b1de7f28098e28c6c4dd7460ab7d1887fe42a301368a0f24"}
Sep 09 16:24:35.736496 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:35.736461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms" event={"ID":"eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50","Type":"ContainerStarted","Data":"08b00582480dcf2975640e6b25e23cb55c4d4315e53fbc70678f7184b83c27ae"}
Sep 09 16:24:35.752074 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:24:35.752025    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-769d969b5b-w77ms" podStartSLOduration=1.816042812 podStartE2EDuration="4.752010974s" totalImagesPullingTime="2.935968162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:24:35.751317374 +0000 UTC m=+655.700898222" watchObservedRunningTime="2026-09-09 16:24:35.752010974 +0000 UTC m=+655.701591820"
Sep 09 16:26:13.271788 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.271759    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"]
Sep 09 16:26:13.274740 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.274721    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.280336 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.280310    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 16:26:13.280449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.280352    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 16:26:13.281095 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.281080    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 16:26:13.281170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.281109    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6rwjw\""
Sep 09 16:26:13.281170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.281143    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 16:26:13.283982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.283943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"]
Sep 09 16:26:13.409953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.409919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ldh\" (UniqueName: \"kubernetes.io/projected/98249b20-e06f-4107-8aee-96a8fbeaa314-kube-api-access-65ldh\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.410143 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.409992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98249b20-e06f-4107-8aee-96a8fbeaa314-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.410143 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.410029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98249b20-e06f-4107-8aee-96a8fbeaa314-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.510381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.510352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98249b20-e06f-4107-8aee-96a8fbeaa314-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.510559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.510392    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ldh\" (UniqueName: \"kubernetes.io/projected/98249b20-e06f-4107-8aee-96a8fbeaa314-kube-api-access-65ldh\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.510559 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.510442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98249b20-e06f-4107-8aee-96a8fbeaa314-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.511012 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.510992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/98249b20-e06f-4107-8aee-96a8fbeaa314-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.512828 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.512809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/98249b20-e06f-4107-8aee-96a8fbeaa314-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.517761 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.517732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ldh\" (UniqueName: \"kubernetes.io/projected/98249b20-e06f-4107-8aee-96a8fbeaa314-kube-api-access-65ldh\") pod \"kuadrant-console-plugin-fbdd4f445-sc2xk\" (UID: \"98249b20-e06f-4107-8aee-96a8fbeaa314\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.589821 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.589788    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"
Sep 09 16:26:13.858166 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:13.858080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk"]
Sep 09 16:26:13.860390 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:26:13.860357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98249b20_e06f_4107_8aee_96a8fbeaa314.slice/crio-cd986514253dbe012e473f9ba4b0f1fdad95cc411cb2a025ecec6680697a6376 WatchSource:0}: Error finding container cd986514253dbe012e473f9ba4b0f1fdad95cc411cb2a025ecec6680697a6376: Status 404 returned error can't find the container with id cd986514253dbe012e473f9ba4b0f1fdad95cc411cb2a025ecec6680697a6376
Sep 09 16:26:14.124809 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:14.124730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk" event={"ID":"98249b20-e06f-4107-8aee-96a8fbeaa314","Type":"ContainerStarted","Data":"cd986514253dbe012e473f9ba4b0f1fdad95cc411cb2a025ecec6680697a6376"}
Sep 09 16:26:20.159572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:20.159536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk" event={"ID":"98249b20-e06f-4107-8aee-96a8fbeaa314","Type":"ContainerStarted","Data":"9a0fd10e5e5d8687694c434092e7723aaf329f2f5a943e9195b9eb9688e13153"}
Sep 09 16:26:20.174716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:20.174660    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-sc2xk" podStartSLOduration=1.925657953 podStartE2EDuration="7.174643348s" totalImagesPullingTime="5.248985395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:20.17339285 +0000 UTC m=+760.122973699" watchObservedRunningTime="2026-09-09 16:26:20.174643348 +0000 UTC m=+760.124224193"
Sep 09 16:26:25.241822 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.241760    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"]
Sep 09 16:26:25.245771 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.245748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:25.248528 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.248506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dfvhh\""
Sep 09 16:26:25.255540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.255516    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"]
Sep 09 16:26:25.409788 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.409742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzw6v\" (UniqueName: \"kubernetes.io/projected/9e4c6919-ac14-41ed-a9c9-ab26e265267c-kube-api-access-bzw6v\") pod \"limitador-operator-controller-manager-74495c69f8-2wnv6\" (UID: \"9e4c6919-ac14-41ed-a9c9-ab26e265267c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:25.511130 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.511030    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw6v\" (UniqueName: \"kubernetes.io/projected/9e4c6919-ac14-41ed-a9c9-ab26e265267c-kube-api-access-bzw6v\") pod \"limitador-operator-controller-manager-74495c69f8-2wnv6\" (UID: \"9e4c6919-ac14-41ed-a9c9-ab26e265267c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:25.519293 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.519253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw6v\" (UniqueName: \"kubernetes.io/projected/9e4c6919-ac14-41ed-a9c9-ab26e265267c-kube-api-access-bzw6v\") pod \"limitador-operator-controller-manager-74495c69f8-2wnv6\" (UID: \"9e4c6919-ac14-41ed-a9c9-ab26e265267c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:25.557752 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.557714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:25.832979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:25.832839    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"]
Sep 09 16:26:25.835705 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:26:25.835677    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4c6919_ac14_41ed_a9c9_ab26e265267c.slice/crio-3db72decccca0f6eea52d87fc1635766c02309837e53e927fdb4791bae21c422 WatchSource:0}: Error finding container 3db72decccca0f6eea52d87fc1635766c02309837e53e927fdb4791bae21c422: Status 404 returned error can't find the container with id 3db72decccca0f6eea52d87fc1635766c02309837e53e927fdb4791bae21c422
Sep 09 16:26:26.186672 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:26.186591    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6" event={"ID":"9e4c6919-ac14-41ed-a9c9-ab26e265267c","Type":"ContainerStarted","Data":"3db72decccca0f6eea52d87fc1635766c02309837e53e927fdb4791bae21c422"}
Sep 09 16:26:28.197484 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:28.197454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6" event={"ID":"9e4c6919-ac14-41ed-a9c9-ab26e265267c","Type":"ContainerStarted","Data":"6da18956ebdcb89fffa36464fac6015cc8abedcfce36fa39ac9ac221c2d90a63"}
Sep 09 16:26:28.197773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:28.197518    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:26:28.220079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:28.220029    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6" podStartSLOduration=0.938208547 podStartE2EDuration="3.220007895s" totalImagesPullingTime="2.281799348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:26:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:26:28.218573457 +0000 UTC m=+768.168154304" watchObservedRunningTime="2026-09-09 16:26:28.220007895 +0000 UTC m=+768.169588743"
Sep 09 16:26:39.204123 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:26:39.204088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2wnv6"
Sep 09 16:27:09.429423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.429341    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:09.431528 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.431502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:09.433870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.433848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84zp8\""
Sep 09 16:27:09.439477 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.439455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:09.468843 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.468810    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj25\" (UniqueName: \"kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25\") pod \"authorino-7bcfd8f9cd-qf25v\" (UID: \"a102bd97-87ce-4660-b263-b2907b2632d6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:09.569889 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.569856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj25\" (UniqueName: \"kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25\") pod \"authorino-7bcfd8f9cd-qf25v\" (UID: \"a102bd97-87ce-4660-b263-b2907b2632d6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:09.577223 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.577198    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj25\" (UniqueName: \"kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25\") pod \"authorino-7bcfd8f9cd-qf25v\" (UID: \"a102bd97-87ce-4660-b263-b2907b2632d6\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:09.741497 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:09.741409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:10.009728 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:10.009650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:10.013114 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:27:10.013083    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda102bd97_87ce_4660_b263_b2907b2632d6.slice/crio-7942b8c303ba0d5c28632f23d63d339c20090583512acb8845dfe4208333a8d5 WatchSource:0}: Error finding container 7942b8c303ba0d5c28632f23d63d339c20090583512acb8845dfe4208333a8d5: Status 404 returned error can't find the container with id 7942b8c303ba0d5c28632f23d63d339c20090583512acb8845dfe4208333a8d5
Sep 09 16:27:10.388966 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:10.388915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" event={"ID":"a102bd97-87ce-4660-b263-b2907b2632d6","Type":"ContainerStarted","Data":"7942b8c303ba0d5c28632f23d63d339c20090583512acb8845dfe4208333a8d5"}
Sep 09 16:27:13.048067 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:13.048033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:13.406108 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:13.406025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" event={"ID":"a102bd97-87ce-4660-b263-b2907b2632d6","Type":"ContainerStarted","Data":"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"}
Sep 09 16:27:13.420649 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:13.420603    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" podStartSLOduration=1.912459021 podStartE2EDuration="4.420587603s" totalImagesPullingTime="2.508128582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:27:13.41862743 +0000 UTC m=+813.368208274" watchObservedRunningTime="2026-09-09 16:27:13.420587603 +0000 UTC m=+813.370168451"
Sep 09 16:27:14.410561 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:14.410525    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" podUID="a102bd97-87ce-4660-b263-b2907b2632d6" containerName="authorino" containerID="cri-o://33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab" gracePeriod=30
Sep 09 16:27:14.665145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:14.665090    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:14.716408 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:14.716371    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lj25\" (UniqueName: \"kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25\") pod \"a102bd97-87ce-4660-b263-b2907b2632d6\" (UID: \"a102bd97-87ce-4660-b263-b2907b2632d6\") "
Sep 09 16:27:14.718514 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:14.718491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25" (OuterVolumeSpecName: "kube-api-access-4lj25") pod "a102bd97-87ce-4660-b263-b2907b2632d6" (UID: "a102bd97-87ce-4660-b263-b2907b2632d6"). InnerVolumeSpecName "kube-api-access-4lj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:27:14.817019 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:14.816984    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lj25\" (UniqueName: \"kubernetes.io/projected/a102bd97-87ce-4660-b263-b2907b2632d6-kube-api-access-4lj25\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:27:15.415544 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.415509    2559 generic.go:356] "Generic (PLEG): container finished" podID="a102bd97-87ce-4660-b263-b2907b2632d6" containerID="33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab" exitCode=0
Sep 09 16:27:15.415930 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.415548    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v"
Sep 09 16:27:15.415930 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.415597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" event={"ID":"a102bd97-87ce-4660-b263-b2907b2632d6","Type":"ContainerDied","Data":"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"}
Sep 09 16:27:15.415930 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.415639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qf25v" event={"ID":"a102bd97-87ce-4660-b263-b2907b2632d6","Type":"ContainerDied","Data":"7942b8c303ba0d5c28632f23d63d339c20090583512acb8845dfe4208333a8d5"}
Sep 09 16:27:15.415930 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.415656    2559 scope.go:117] "RemoveContainer" containerID="33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"
Sep 09 16:27:15.424066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.424051    2559 scope.go:117] "RemoveContainer" containerID="33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"
Sep 09 16:27:15.424294 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:27:15.424275    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab\": container with ID starting with 33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab not found: ID does not exist" containerID="33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"
Sep 09 16:27:15.424340 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.424302    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab"} err="failed to get container status \"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab\": rpc error: code = NotFound desc = could not find container \"33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab\": container with ID starting with 33538e35440712e55f06da39bae5d8dda28ba4c74c5d24a45d78201ebec149ab not found: ID does not exist"
Sep 09 16:27:15.434713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.434694    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:15.436885 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:15.436865    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qf25v"]
Sep 09 16:27:16.530969 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:27:16.530924    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a102bd97-87ce-4660-b263-b2907b2632d6" path="/var/lib/kubelet/pods/a102bd97-87ce-4660-b263-b2907b2632d6/volumes"
Sep 09 16:28:41.380393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:41.380366    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:28:41.380951 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:41.380935    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:28:56.794176 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.794143    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"]
Sep 09 16:28:56.795506 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.794596    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a102bd97-87ce-4660-b263-b2907b2632d6" containerName="authorino"
Sep 09 16:28:56.795506 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.794612    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a102bd97-87ce-4660-b263-b2907b2632d6" containerName="authorino"
Sep 09 16:28:56.795506 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.794678    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a102bd97-87ce-4660-b263-b2907b2632d6" containerName="authorino"
Sep 09 16:28:56.796397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.796381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.798902 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.798883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 16:28:56.799024 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.798883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 16:28:56.799846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.799824    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x7dhn\""
Sep 09 16:28:56.799982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.799942    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 16:28:56.805778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.805757    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"]
Sep 09 16:28:56.830172 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.830270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.830270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfd5\" (UniqueName: \"kubernetes.io/projected/f9c2090d-47c1-4dcb-9c73-109b191756d6-kube-api-access-tsfd5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.830349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2090d-47c1-4dcb-9c73-109b191756d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.830349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.830419 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.830375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931167 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931167 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931167 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfd5\" (UniqueName: \"kubernetes.io/projected/f9c2090d-47c1-4dcb-9c73-109b191756d6-kube-api-access-tsfd5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2090d-47c1-4dcb-9c73-109b191756d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931518 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931573 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.931619 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.931587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.933466 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.933447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9c2090d-47c1-4dcb-9c73-109b191756d6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.933568 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.933551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9c2090d-47c1-4dcb-9c73-109b191756d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:56.938458 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:56.938435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfd5\" (UniqueName: \"kubernetes.io/projected/f9c2090d-47c1-4dcb-9c73-109b191756d6-kube-api-access-tsfd5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-m4gnz\" (UID: \"f9c2090d-47c1-4dcb-9c73-109b191756d6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:57.107019 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:57.106922    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:28:57.375199 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:57.375174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"]
Sep 09 16:28:57.377865 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:28:57.377828    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c2090d_47c1_4dcb_9c73_109b191756d6.slice/crio-506624f4e7bea07a932d9cf3d58b601ca5012f1af605dad69d4bea094417de74 WatchSource:0}: Error finding container 506624f4e7bea07a932d9cf3d58b601ca5012f1af605dad69d4bea094417de74: Status 404 returned error can't find the container with id 506624f4e7bea07a932d9cf3d58b601ca5012f1af605dad69d4bea094417de74
Sep 09 16:28:57.860298 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:57.860237    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz" event={"ID":"f9c2090d-47c1-4dcb-9c73-109b191756d6","Type":"ContainerStarted","Data":"506624f4e7bea07a932d9cf3d58b601ca5012f1af605dad69d4bea094417de74"}
Sep 09 16:28:58.866326 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:58.866294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz" event={"ID":"f9c2090d-47c1-4dcb-9c73-109b191756d6","Type":"ContainerStarted","Data":"c8d5c4224c4f9b3a841c766d4477ce1774636ea184ab7843b96ec9193e88eb5e"}
Sep 09 16:28:58.882998 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:28:58.882939    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz" podStartSLOduration=1.895885422 podStartE2EDuration="2.882924108s" totalImagesPullingTime="987.038686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:28:58.88151937 +0000 UTC m=+918.831100227" watchObservedRunningTime="2026-09-09 16:28:58.882924108 +0000 UTC m=+918.832504965"
Sep 09 16:29:07.107456 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:07.107408    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:29:07.108020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:07.107484    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:29:07.120088 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:07.120058    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:29:07.918952 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:07.918922    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-m4gnz"
Sep 09 16:29:37.391749 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.391713    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"]
Sep 09 16:29:37.397545 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.397523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.400134 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.400108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 16:29:37.400800 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.400775    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"]
Sep 09 16:29:37.572522 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.572522 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx97\" (UniqueName: \"kubernetes.io/projected/cfeca21e-b5de-4dd9-bb17-fb306babebbf-kube-api-access-ktx97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.572729 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.572729 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.572729 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.572729 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.572654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.673987 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.673892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.673987 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.673937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.673987 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.673954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.673992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.674045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.674066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx97\" (UniqueName: \"kubernetes.io/projected/cfeca21e-b5de-4dd9-bb17-fb306babebbf-kube-api-access-ktx97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674424 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.674400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674472 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.674436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.674507 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.674496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.675998 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.675971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfeca21e-b5de-4dd9-bb17-fb306babebbf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.676393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.676374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfeca21e-b5de-4dd9-bb17-fb306babebbf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.681530 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.681506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx97\" (UniqueName: \"kubernetes.io/projected/cfeca21e-b5de-4dd9-bb17-fb306babebbf-kube-api-access-ktx97\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tblx9\" (UID: \"cfeca21e-b5de-4dd9-bb17-fb306babebbf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.709400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.709366    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:37.977024 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.977000    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"]
Sep 09 16:29:37.979077 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:29:37.979051    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeca21e_b5de_4dd9_bb17_fb306babebbf.slice/crio-0ab2d9463503f8273296b3e5d37de8b98275257c6f51e020313f9dc370971b16 WatchSource:0}: Error finding container 0ab2d9463503f8273296b3e5d37de8b98275257c6f51e020313f9dc370971b16: Status 404 returned error can't find the container with id 0ab2d9463503f8273296b3e5d37de8b98275257c6f51e020313f9dc370971b16
Sep 09 16:29:37.980485 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:37.980467    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:29:38.044702 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:38.044667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9" event={"ID":"cfeca21e-b5de-4dd9-bb17-fb306babebbf","Type":"ContainerStarted","Data":"0ab2d9463503f8273296b3e5d37de8b98275257c6f51e020313f9dc370971b16"}
Sep 09 16:29:39.050181 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:39.050144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9" event={"ID":"cfeca21e-b5de-4dd9-bb17-fb306babebbf","Type":"ContainerStarted","Data":"cbeb6086277e7ff859568678742b4b84c9de41e3d0d776d5e76ee6c4a3acd424"}
Sep 09 16:29:39.065610 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:39.065562    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9" podStartSLOduration=1.853265819 podStartE2EDuration="2.065547919s" totalImagesPullingTime="212.2821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:39.063840244 +0000 UTC m=+959.013421113" watchObservedRunningTime="2026-09-09 16:29:39.065547919 +0000 UTC m=+959.015128766"
Sep 09 16:29:47.710121 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:47.710081    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:47.710574 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:47.710163    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:47.722507 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:47.722486    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:48.103028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:48.102993    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tblx9"
Sep 09 16:29:51.779137 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.779101    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"]
Sep 09 16:29:51.782804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.782780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.785212 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.785194    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 16:29:51.792669 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.792649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"]
Sep 09 16:29:51.886871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.886840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2867\" (UniqueName: \"kubernetes.io/projected/bfdffb26-b427-4c61-8e1c-807cb9e11c24-kube-api-access-k2867\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.887035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.886887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.887035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.886980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.887126 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.887044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.887126 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.887066    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.887126 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.887094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988329 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2867\" (UniqueName: \"kubernetes.io/projected/bfdffb26-b427-4c61-8e1c-807cb9e11c24-kube-api-access-k2867\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988750 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988820 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.988864 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.988825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.990727 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.990705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdffb26-b427-4c61-8e1c-807cb9e11c24-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.990910 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.990895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdffb26-b427-4c61-8e1c-807cb9e11c24-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:51.995796 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:51.995763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2867\" (UniqueName: \"kubernetes.io/projected/bfdffb26-b427-4c61-8e1c-807cb9e11c24-kube-api-access-k2867\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vngp9\" (UID: \"bfdffb26-b427-4c61-8e1c-807cb9e11c24\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:52.094458 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:52.094430    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:29:52.363935 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:52.363908    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"]
Sep 09 16:29:52.366131 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:29:52.366105    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdffb26_b427_4c61_8e1c_807cb9e11c24.slice/crio-df45e8b41c25c7e38e26571f54c9432b4892ce7d669db14970bc11eb8c26cbc5 WatchSource:0}: Error finding container df45e8b41c25c7e38e26571f54c9432b4892ce7d669db14970bc11eb8c26cbc5: Status 404 returned error can't find the container with id df45e8b41c25c7e38e26571f54c9432b4892ce7d669db14970bc11eb8c26cbc5
Sep 09 16:29:53.117971 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:53.117930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9" event={"ID":"bfdffb26-b427-4c61-8e1c-807cb9e11c24","Type":"ContainerStarted","Data":"2c3480d21a2ade12e984204dcc10924c7956352a7dc23d6a88193f2efa491223"}
Sep 09 16:29:53.118334 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:53.117997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9" event={"ID":"bfdffb26-b427-4c61-8e1c-807cb9e11c24","Type":"ContainerStarted","Data":"df45e8b41c25c7e38e26571f54c9432b4892ce7d669db14970bc11eb8c26cbc5"}
Sep 09 16:29:53.134005 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:29:53.133946    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9" podStartSLOduration=1.900846804 podStartE2EDuration="2.133930324s" totalImagesPullingTime="233.08352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:29:53.132457633 +0000 UTC m=+973.082038481" watchObservedRunningTime="2026-09-09 16:29:53.133930324 +0000 UTC m=+973.083511170"
Sep 09 16:30:02.095068 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:30:02.095027    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:30:02.095483 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:30:02.095097    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:30:02.107629 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:30:02.107600    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:30:02.170353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:30:02.170314    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vngp9"
Sep 09 16:31:56.129307 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.129213    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b544d5b74-4rnmr"]
Sep 09 16:31:56.132726 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.132700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.135927 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.135897    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-84zp8\""
Sep 09 16:31:56.135927 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.135920    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 16:31:56.148667 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.148638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b544d5b74-4rnmr"]
Sep 09 16:31:56.257334 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.257291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blxj\" (UniqueName: \"kubernetes.io/projected/b93bbb15-4526-42eb-b1ba-f789de04f939-kube-api-access-8blxj\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.257509 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.257405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b93bbb15-4526-42eb-b1ba-f789de04f939-tls-cert\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.358293 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.358251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blxj\" (UniqueName: \"kubernetes.io/projected/b93bbb15-4526-42eb-b1ba-f789de04f939-kube-api-access-8blxj\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.358485 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.358328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b93bbb15-4526-42eb-b1ba-f789de04f939-tls-cert\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.360763 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.360731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b93bbb15-4526-42eb-b1ba-f789de04f939-tls-cert\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.365930 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.365901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blxj\" (UniqueName: \"kubernetes.io/projected/b93bbb15-4526-42eb-b1ba-f789de04f939-kube-api-access-8blxj\") pod \"authorino-5b544d5b74-4rnmr\" (UID: \"b93bbb15-4526-42eb-b1ba-f789de04f939\") " pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.442124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.442022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b544d5b74-4rnmr"
Sep 09 16:31:56.714181 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:56.714105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b544d5b74-4rnmr"]
Sep 09 16:31:56.717527 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:31:56.717499    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93bbb15_4526_42eb_b1ba_f789de04f939.slice/crio-090f4d1d44f0c6dee569a587760b2dd5e31c43cabf2081e7bdca01b0e2c89e06 WatchSource:0}: Error finding container 090f4d1d44f0c6dee569a587760b2dd5e31c43cabf2081e7bdca01b0e2c89e06: Status 404 returned error can't find the container with id 090f4d1d44f0c6dee569a587760b2dd5e31c43cabf2081e7bdca01b0e2c89e06
Sep 09 16:31:57.704597 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:57.704566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b544d5b74-4rnmr" event={"ID":"b93bbb15-4526-42eb-b1ba-f789de04f939","Type":"ContainerStarted","Data":"090f4d1d44f0c6dee569a587760b2dd5e31c43cabf2081e7bdca01b0e2c89e06"}
Sep 09 16:31:58.711513 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:58.711481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b544d5b74-4rnmr" event={"ID":"b93bbb15-4526-42eb-b1ba-f789de04f939","Type":"ContainerStarted","Data":"e477b77e053c79597c8c81271390aa738b94b08d97ac53693ef1e71d77706388"}
Sep 09 16:31:58.729867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:31:58.729807    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b544d5b74-4rnmr" podStartSLOduration=1.772321832 podStartE2EDuration="2.729788038s" totalImagesPullingTime="957.466206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:31:58.728169857 +0000 UTC m=+1098.677750705" watchObservedRunningTime="2026-09-09 16:31:58.729788038 +0000 UTC m=+1098.679368885"
Sep 09 16:32:39.041790 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.041755    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:32:39.045201 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.045181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.047478 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.047453    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e94c211c\""
Sep 09 16:32:39.047592 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.047526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e94c211c-dockercfg-m6gds\""
Sep 09 16:32:39.055518 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.055494    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:32:39.066648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.066625    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:32:39.069943 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.069926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.081875 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.081853    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:32:39.130458 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.130427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w82fw\" (UniqueName: \"kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.130604 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.130496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64d5\" (UniqueName: \"kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.130604 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.130580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.130706 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.130635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.231412 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.231380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w82fw\" (UniqueName: \"kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.231566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.231426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64d5\" (UniqueName: \"kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.231566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.231462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.231566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.231488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.232156 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.232133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.232255 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.232192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.239429 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.239402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64d5\" (UniqueName: \"kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5\") pod \"payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") " pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.239526 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.239500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w82fw\" (UniqueName: \"kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw\") pod \"payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.355747 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.355666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:39.380360 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.380335    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:39.632668 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.632643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:32:39.659029 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.659008    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:32:39.661210 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:32:39.661178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8265b519_530b_4dfa_af87_6e32feea2b06.slice/crio-7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32 WatchSource:0}: Error finding container 7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32: Status 404 returned error can't find the container with id 7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32
Sep 09 16:32:39.908531 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.908430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" event={"ID":"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d","Type":"ContainerStarted","Data":"1b20ad7099db495c957f0ffcc3e84073e9be68562b37292549cf8e504b00e8b5"}
Sep 09 16:32:39.909612 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:39.909588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" event={"ID":"8265b519-530b-4dfa-af87-6e32feea2b06","Type":"ContainerStarted","Data":"7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32"}
Sep 09 16:32:40.075062 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.075029    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:32:40.079543 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.079523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.082092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.082071    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-05e496\""
Sep 09 16:32:40.082204 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.082158    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-05e496-dockercfg-8qv5w\""
Sep 09 16:32:40.087499 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.087476    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:32:40.142516 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.142487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcjt\" (UniqueName: \"kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.142665 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.142542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.244671 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.243869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.244671 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.244109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcjt\" (UniqueName: \"kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.244889 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.244805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.256770 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.256718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcjt\" (UniqueName: \"kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt\") pod \"payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.391205 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.391115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:40.689996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.689852    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:32:40.702904 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:32:40.702866    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbb9cdf_099d_49b5_b9fc_c1c3edf09e44.slice/crio-ec4c4f3e01168b1f6e7c54e8449be11e134eaa5e07edb22efcef4fe1280f565f WatchSource:0}: Error finding container ec4c4f3e01168b1f6e7c54e8449be11e134eaa5e07edb22efcef4fe1280f565f: Status 404 returned error can't find the container with id ec4c4f3e01168b1f6e7c54e8449be11e134eaa5e07edb22efcef4fe1280f565f
Sep 09 16:32:40.919335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:40.919290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" event={"ID":"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44","Type":"ContainerStarted","Data":"ec4c4f3e01168b1f6e7c54e8449be11e134eaa5e07edb22efcef4fe1280f565f"}
Sep 09 16:32:41.925031 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.924911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" event={"ID":"8265b519-530b-4dfa-af87-6e32feea2b06","Type":"ContainerStarted","Data":"d8475e5aa0de1972c7abe0de492a804261f6be3a20a58300ff8682905bb31c08"}
Sep 09 16:32:41.925031 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.925000    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:41.926443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.926421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" event={"ID":"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d","Type":"ContainerStarted","Data":"7eafadc9b09c01406cbe85504836e3db4c625a8d3d6a6abe16f5b1d169b901bf"}
Sep 09 16:32:41.926558 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.926544    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:41.939531 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.939465    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" podStartSLOduration=0.935160439 podStartE2EDuration="2.939442077s" totalImagesPullingTime="2.004281638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:41.938867324 +0000 UTC m=+1141.888448175" watchObservedRunningTime="2026-09-09 16:32:41.939442077 +0000 UTC m=+1141.889022925"
Sep 09 16:32:41.955307 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:41.955249    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" podStartSLOduration=0.919664093 podStartE2EDuration="2.955229051s" totalImagesPullingTime="2.035564958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:41.952844594 +0000 UTC m=+1141.902425455" watchObservedRunningTime="2026-09-09 16:32:41.955229051 +0000 UTC m=+1141.904809898"
Sep 09 16:32:42.933720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:42.933674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" event={"ID":"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44","Type":"ContainerStarted","Data":"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"}
Sep 09 16:32:42.949752 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:42.949701    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" podStartSLOduration=1.701714569 podStartE2EDuration="2.949680043s" totalImagesPullingTime="1.247965474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:32:42.948691703 +0000 UTC m=+1142.898272552" watchObservedRunningTime="2026-09-09 16:32:42.949680043 +0000 UTC m=+1142.899260889"
Sep 09 16:32:43.938972 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:43.938929    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:52.935495 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:52.935464    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:32:52.935862 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:52.935542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:32:54.946043 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:54.946005    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:32:56.148037 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.148002    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"]
Sep 09 16:32:56.151361 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.151343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:32:56.153814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.153794    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:32:56.153906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.153813    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jz5ff\""
Sep 09 16:32:56.159363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.159335    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"]
Sep 09 16:32:56.287269 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.287238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg\" (UID: \"2fcf9002-753a-4323-a973-36b648f8af2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:32:56.388412 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.388349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg\" (UID: \"2fcf9002-753a-4323-a973-36b648f8af2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:32:56.389020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.388997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg\" (UID: \"2fcf9002-753a-4323-a973-36b648f8af2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:32:56.462707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.462621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:32:56.735641 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:56.735614    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"]
Sep 09 16:32:56.741078 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:32:56.741033    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcf9002_753a_4323_a973_36b648f8af2c.slice/crio-184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69 WatchSource:0}: Error finding container 184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69: Status 404 returned error can't find the container with id 184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69
Sep 09 16:32:57.005539 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:57.005450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg" event={"ID":"2fcf9002-753a-4323-a973-36b648f8af2c","Type":"ContainerStarted","Data":"184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69"}
Sep 09 16:32:58.449778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.449736    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:32:58.457243 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.457141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.459712 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.459681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-34141f\""
Sep 09 16:32:58.460343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.460321    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-34141f-dockercfg-wdlvd\""
Sep 09 16:32:58.462125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.462100    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:32:58.466247 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.466191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:32:58.466392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.466302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.482935 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.480440    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:32:58.608566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.608528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqlk\" (UniqueName: \"kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.608746 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.608697    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.608814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.608743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.608814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.608769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4dz\" (UniqueName: \"kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.710392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.710304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqlk\" (UniqueName: \"kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.710540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.710407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.710540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.710432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.710540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.710457    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4dz\" (UniqueName: \"kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.711291 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.711266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.711405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.711356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.721682 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.721650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqlk\" (UniqueName: \"kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk\") pod \"payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.722952 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.722920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4dz\" (UniqueName: \"kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz\") pod \"payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:58.772058 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.772006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:32:58.791500 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:58.791462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:32:59.348275 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:59.348241    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:32:59.350554 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:32:59.350524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:32:59.351179 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:32:59.351156    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61dfbec_f5be_4e32_945f_5515d11d373e.slice/crio-f6bfea6f25ce4c4c85cd9c6b8baad845e0078d672484c2131c133b1501f48d01 WatchSource:0}: Error finding container f6bfea6f25ce4c4c85cd9c6b8baad845e0078d672484c2131c133b1501f48d01: Status 404 returned error can't find the container with id f6bfea6f25ce4c4c85cd9c6b8baad845e0078d672484c2131c133b1501f48d01
Sep 09 16:32:59.352467 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:32:59.352448    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3277f19_f369_4a46_9cd2_31e67e5fe9da.slice/crio-dd82d345820f67d9d779e57c78f71e475eea3fc5bca77335ccd8b45088215037 WatchSource:0}: Error finding container dd82d345820f67d9d779e57c78f71e475eea3fc5bca77335ccd8b45088215037: Status 404 returned error can't find the container with id dd82d345820f67d9d779e57c78f71e475eea3fc5bca77335ccd8b45088215037
Sep 09 16:33:00.025692 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.025647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" event={"ID":"b3277f19-f369-4a46-9cd2-31e67e5fe9da","Type":"ContainerStarted","Data":"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"}
Sep 09 16:33:00.026202 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.025716    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:33:00.026202 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.025738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" event={"ID":"b3277f19-f369-4a46-9cd2-31e67e5fe9da","Type":"ContainerStarted","Data":"dd82d345820f67d9d779e57c78f71e475eea3fc5bca77335ccd8b45088215037"}
Sep 09 16:33:00.027495 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.027465    2559 generic.go:356] "Generic (PLEG): container finished" podID="2fcf9002-753a-4323-a973-36b648f8af2c" containerID="9aeaf71993da48ff56ddb06ffa1731e1144f5f1236f194423d4faba5a668fe3f" exitCode=0
Sep 09 16:33:00.027633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.027558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg" event={"ID":"2fcf9002-753a-4323-a973-36b648f8af2c","Type":"ContainerDied","Data":"9aeaf71993da48ff56ddb06ffa1731e1144f5f1236f194423d4faba5a668fe3f"}
Sep 09 16:33:00.029266 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.029240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" event={"ID":"e61dfbec-f5be-4e32-945f-5515d11d373e","Type":"ContainerStarted","Data":"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"}
Sep 09 16:33:00.029377 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.029280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" event={"ID":"e61dfbec-f5be-4e32-945f-5515d11d373e","Type":"ContainerStarted","Data":"f6bfea6f25ce4c4c85cd9c6b8baad845e0078d672484c2131c133b1501f48d01"}
Sep 09 16:33:00.029377 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.029302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:33:00.041839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.041787    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" podStartSLOduration=2.041768851 podStartE2EDuration="2.041768851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:00.040591045 +0000 UTC m=+1159.990171905" watchObservedRunningTime="2026-09-09 16:33:00.041768851 +0000 UTC m=+1159.991349747"
Sep 09 16:33:00.076011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:00.075865    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" podStartSLOduration=2.075845628 podStartE2EDuration="2.075845628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:00.073728227 +0000 UTC m=+1160.023309075" watchObservedRunningTime="2026-09-09 16:33:00.075845628 +0000 UTC m=+1160.025426475"
Sep 09 16:33:01.163209 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:01.163188    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:33:01.334018 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:01.333913    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca\") pod \"2fcf9002-753a-4323-a973-36b648f8af2c\" (UID: \"2fcf9002-753a-4323-a973-36b648f8af2c\") "
Sep 09 16:33:01.334249 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:01.334227    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fcf9002-753a-4323-a973-36b648f8af2c" (UID: "2fcf9002-753a-4323-a973-36b648f8af2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:01.434724 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:01.434694    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fcf9002-753a-4323-a973-36b648f8af2c-maas-api-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:02.040725 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:02.040701    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"
Sep 09 16:33:02.040886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:02.040697    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg" event={"ID":"2fcf9002-753a-4323-a973-36b648f8af2c","Type":"ContainerDied","Data":"184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69"}
Sep 09 16:33:02.040886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:02.040825    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184a8f5c665c801f2b7672e480c9c6563fd815a3f31748a8feaf88f5fd3f1e69"
Sep 09 16:33:06.914889 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:06.914800    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:33:06.915364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:06.915050    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" podUID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" containerName="payload-processing" containerID="cri-o://7eafadc9b09c01406cbe85504836e3db4c625a8d3d6a6abe16f5b1d169b901bf" gracePeriod=30
Sep 09 16:33:06.920762 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:06.920716    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:33:06.921321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:06.921279    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" podUID="8265b519-530b-4dfa-af87-6e32feea2b06" containerName="payload-pre-processing" containerID="cri-o://d8475e5aa0de1972c7abe0de492a804261f6be3a20a58300ff8682905bb31c08" gracePeriod=30
Sep 09 16:33:07.055707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.055669    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"]
Sep 09 16:33:07.058291 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.057755    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e94c211c-jgxmg"]
Sep 09 16:33:07.067173 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.067137    2559 generic.go:356] "Generic (PLEG): container finished" podID="8265b519-530b-4dfa-af87-6e32feea2b06" containerID="d8475e5aa0de1972c7abe0de492a804261f6be3a20a58300ff8682905bb31c08" exitCode=0
Sep 09 16:33:07.067367 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.067192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" event={"ID":"8265b519-530b-4dfa-af87-6e32feea2b06","Type":"ContainerDied","Data":"d8475e5aa0de1972c7abe0de492a804261f6be3a20a58300ff8682905bb31c08"}
Sep 09 16:33:07.069057 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.069007    2559 generic.go:356] "Generic (PLEG): container finished" podID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" containerID="7eafadc9b09c01406cbe85504836e3db4c625a8d3d6a6abe16f5b1d169b901bf" exitCode=0
Sep 09 16:33:07.069215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.069089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" event={"ID":"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d","Type":"ContainerDied","Data":"7eafadc9b09c01406cbe85504836e3db4c625a8d3d6a6abe16f5b1d169b901bf"}
Sep 09 16:33:07.169441 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.169370    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:33:07.198452 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.198420    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:33:07.281102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.281068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume\") pod \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") "
Sep 09 16:33:07.281280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.281145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t64d5\" (UniqueName: \"kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5\") pod \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\" (UID: \"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d\") "
Sep 09 16:33:07.281453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.281428    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" (UID: "5d19d41f-1b19-4b1a-b1bf-e49eee786a7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:07.283405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.283373    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5" (OuterVolumeSpecName: "kube-api-access-t64d5") pod "5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" (UID: "5d19d41f-1b19-4b1a-b1bf-e49eee786a7d"). InnerVolumeSpecName "kube-api-access-t64d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:07.382469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.382417    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w82fw\" (UniqueName: \"kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw\") pod \"8265b519-530b-4dfa-af87-6e32feea2b06\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") "
Sep 09 16:33:07.382469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.382477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume\") pod \"8265b519-530b-4dfa-af87-6e32feea2b06\" (UID: \"8265b519-530b-4dfa-af87-6e32feea2b06\") "
Sep 09 16:33:07.382731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.382620    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64d5\" (UniqueName: \"kubernetes.io/projected/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-kube-api-access-t64d5\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.382731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.382632    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.382882 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.382856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8265b519-530b-4dfa-af87-6e32feea2b06" (UID: "8265b519-530b-4dfa-af87-6e32feea2b06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:07.384698 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.384672    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw" (OuterVolumeSpecName: "kube-api-access-w82fw") pod "8265b519-530b-4dfa-af87-6e32feea2b06" (UID: "8265b519-530b-4dfa-af87-6e32feea2b06"). InnerVolumeSpecName "kube-api-access-w82fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:07.483432 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.483335    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w82fw\" (UniqueName: \"kubernetes.io/projected/8265b519-530b-4dfa-af87-6e32feea2b06-kube-api-access-w82fw\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:07.483432 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:07.483369    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8265b519-530b-4dfa-af87-6e32feea2b06-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:08.075592 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.075557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"
Sep 09 16:33:08.076046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.075588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz" event={"ID":"8265b519-530b-4dfa-af87-6e32feea2b06","Type":"ContainerDied","Data":"7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32"}
Sep 09 16:33:08.076046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.075691    2559 scope.go:117] "RemoveContainer" containerID="d8475e5aa0de1972c7abe0de492a804261f6be3a20a58300ff8682905bb31c08"
Sep 09 16:33:08.077196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.077068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn" event={"ID":"5d19d41f-1b19-4b1a-b1bf-e49eee786a7d","Type":"ContainerDied","Data":"1b20ad7099db495c957f0ffcc3e84073e9be68562b37292549cf8e504b00e8b5"}
Sep 09 16:33:08.077196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.077116    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"
Sep 09 16:33:08.086321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.086155    2559 scope.go:117] "RemoveContainer" containerID="7eafadc9b09c01406cbe85504836e3db4c625a8d3d6a6abe16f5b1d169b901bf"
Sep 09 16:33:08.101003 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.100970    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:33:08.107079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.107038    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e94c211c-7b4cdff55d-cf5vz"]
Sep 09 16:33:08.117245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.117216    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:33:08.120678 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:08.120639    2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8265b519_530b_4dfa_af87_6e32feea2b06.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d19d41f_1b19_4b1a_b1bf_e49eee786a7d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8265b519_530b_4dfa_af87_6e32feea2b06.slice/crio-7372d8576cd9a099511638732708c581dcf261a2a328ca6cea10b36caade9b32\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d19d41f_1b19_4b1a_b1bf_e49eee786a7d.slice/crio-1b20ad7099db495c957f0ffcc3e84073e9be68562b37292549cf8e504b00e8b5\": RecentStats: unable to find data in memory cache]"
Sep 09 16:33:08.121002 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.120982    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e94c211c-77b6d6c559-s79mn"]
Sep 09 16:33:08.531804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.531770    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fcf9002-753a-4323-a973-36b648f8af2c" path="/var/lib/kubelet/pods/2fcf9002-753a-4323-a973-36b648f8af2c/volumes"
Sep 09 16:33:08.532170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.532156    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" path="/var/lib/kubelet/pods/5d19d41f-1b19-4b1a-b1bf-e49eee786a7d/volumes"
Sep 09 16:33:08.532462 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:08.532450    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8265b519-530b-4dfa-af87-6e32feea2b06" path="/var/lib/kubelet/pods/8265b519-530b-4dfa-af87-6e32feea2b06/volumes"
Sep 09 16:33:11.036052 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.036021    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:33:11.036424 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.036089    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:33:11.667389 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667356    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:11.667727 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667714    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8265b519-530b-4dfa-af87-6e32feea2b06" containerName="payload-pre-processing"
Sep 09 16:33:11.667789 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667729    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8265b519-530b-4dfa-af87-6e32feea2b06" containerName="payload-pre-processing"
Sep 09 16:33:11.667789 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667738    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" containerName="payload-processing"
Sep 09 16:33:11.667789 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" containerName="payload-processing"
Sep 09 16:33:11.667789 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667758    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcf9002-753a-4323-a973-36b648f8af2c" containerName="revoke-keys"
Sep 09 16:33:11.667789 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667764    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcf9002-753a-4323-a973-36b648f8af2c" containerName="revoke-keys"
Sep 09 16:33:11.667953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667816    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d19d41f-1b19-4b1a-b1bf-e49eee786a7d" containerName="payload-processing"
Sep 09 16:33:11.667953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667823    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8265b519-530b-4dfa-af87-6e32feea2b06" containerName="payload-pre-processing"
Sep 09 16:33:11.667953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.667829    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fcf9002-753a-4323-a973-36b648f8af2c" containerName="revoke-keys"
Sep 09 16:33:11.670774 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.670755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.679401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.674393    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9e0f78a3-serving-cert\""
Sep 09 16:33:11.679401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.674722    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 16:33:11.679401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.679198    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:33:11.679401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.679230    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k96v8\""
Sep 09 16:33:11.679401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.679293    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 16:33:11.683692 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.683667    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:11.817592 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.817565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.817707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.817604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.817707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.817678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.817795 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.817715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2x5\" (UniqueName: \"kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.919058 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.919001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.919058 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.919044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2x5\" (UniqueName: \"kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.919197 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.919083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.919197 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.919106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.919273 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:11.919198    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-serving-cert: secret "maas-api-e2e-migrate-9e0f78a3-serving-cert" not found
Sep 09 16:33:11.919273 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:11.919249    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls podName:8b2b12e1-1f4f-443c-8273-7a3cbb99d04b nodeName:}" failed. No retries permitted until 2026-09-09 16:33:12.419233805 +0000 UTC m=+1172.368814630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls") pod "maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" (UID: "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b") : secret "maas-api-e2e-migrate-9e0f78a3-serving-cert" not found
Sep 09 16:33:11.919713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.919692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.921413 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.921394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:11.929359 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:11.929341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2x5\" (UniqueName: \"kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:12.292227 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.292199    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:12.295680 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.295661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.298156 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.298137    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9e0f78a3\""
Sep 09 16:33:12.298218 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.298174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9e0f78a3-dockercfg-w7scz\""
Sep 09 16:33:12.305366 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.305345    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:12.338615 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.338593    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:12.341745 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.341731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.351720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.351699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:12.423934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.423913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.424046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.423950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.424046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.424033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.424120 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.424071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:12.424120 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.424092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgdh\" (UniqueName: \"kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.426381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.426359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") pod \"maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:12.524676 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.524652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgdh\" (UniqueName: \"kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.524760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.524697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.524760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.524737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.524843 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.524778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.525476 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.525450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.525527 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.525487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.532399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.532376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz\") pod \"payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") " pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.532500 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.532489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgdh\" (UniqueName: \"kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh\") pod \"payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.590302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.590254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:12.607048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.607029    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:12.651431 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.651404    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:12.869166 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.869144    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:12.871232 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:12.871203    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2b12e1_1f4f_443c_8273_7a3cbb99d04b.slice/crio-d343dcff520ae2860adbdeb178e1140018e4bdccecf914d89c08d3519f80e929 WatchSource:0}: Error finding container d343dcff520ae2860adbdeb178e1140018e4bdccecf914d89c08d3519f80e929: Status 404 returned error can't find the container with id d343dcff520ae2860adbdeb178e1140018e4bdccecf914d89c08d3519f80e929
Sep 09 16:33:12.892400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.892379    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:12.894604 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:12.894583    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa859cc_4b86_4170_bab6_4a78f778931f.slice/crio-3285c8f9e87f97f6130b366551318167b398983394a86ad21edd63efb7769bfd WatchSource:0}: Error finding container 3285c8f9e87f97f6130b366551318167b398983394a86ad21edd63efb7769bfd: Status 404 returned error can't find the container with id 3285c8f9e87f97f6130b366551318167b398983394a86ad21edd63efb7769bfd
Sep 09 16:33:12.938599 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:12.938527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:12.941276 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:12.941249    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130d2291_27af_4ac2_bbd0_8f2483d4e22d.slice/crio-32645d697d23d62ad83abc0a032e41a877f4a82c4d95ca7c7d1adc815d1f70e5 WatchSource:0}: Error finding container 32645d697d23d62ad83abc0a032e41a877f4a82c4d95ca7c7d1adc815d1f70e5: Status 404 returned error can't find the container with id 32645d697d23d62ad83abc0a032e41a877f4a82c4d95ca7c7d1adc815d1f70e5
Sep 09 16:33:13.107588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.107551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" event={"ID":"4fa859cc-4b86-4170-bab6-4a78f778931f","Type":"ContainerStarted","Data":"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"}
Sep 09 16:33:13.107755 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.107609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" event={"ID":"4fa859cc-4b86-4170-bab6-4a78f778931f","Type":"ContainerStarted","Data":"3285c8f9e87f97f6130b366551318167b398983394a86ad21edd63efb7769bfd"}
Sep 09 16:33:13.107755 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.107638    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:13.109103 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.109076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" event={"ID":"130d2291-27af-4ac2-bbd0-8f2483d4e22d","Type":"ContainerStarted","Data":"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"}
Sep 09 16:33:13.109203 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.109122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" event={"ID":"130d2291-27af-4ac2-bbd0-8f2483d4e22d","Type":"ContainerStarted","Data":"32645d697d23d62ad83abc0a032e41a877f4a82c4d95ca7c7d1adc815d1f70e5"}
Sep 09 16:33:13.109203 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.109144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:13.110214 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.110194    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" event={"ID":"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b","Type":"ContainerStarted","Data":"d343dcff520ae2860adbdeb178e1140018e4bdccecf914d89c08d3519f80e929"}
Sep 09 16:33:13.123558 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.123485    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" podStartSLOduration=1.1234714239999999 podStartE2EDuration="1.123471424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:13.121942405 +0000 UTC m=+1173.071523252" watchObservedRunningTime="2026-09-09 16:33:13.123471424 +0000 UTC m=+1173.073052271"
Sep 09 16:33:13.137132 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.137096    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" podStartSLOduration=1.137083834 podStartE2EDuration="1.137083834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:13.135403293 +0000 UTC m=+1173.084984141" watchObservedRunningTime="2026-09-09 16:33:13.137083834 +0000 UTC m=+1173.086664680"
Sep 09 16:33:13.385633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.385550    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:33:13.389094 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.389074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.392200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.392180    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-563e2ddd-kserve-self-signed-certs\""
Sep 09 16:33:13.397179 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.397161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:33:13.432145 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.432246 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtkbp\" (UniqueName: \"kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.432246 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.432315 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.432367 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.432417 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.432393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533108 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtkbp\" (UniqueName: \"kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533123    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533527 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533576    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.533811 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.533786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.535936 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.535909    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.536380 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.536356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.540505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.540480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtkbp\" (UniqueName: \"kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp\") pod \"e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") " pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.698997 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.698909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:13.977763 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:13.977684    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:33:13.981301 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:13.981259    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3bd2c33_4e4f_4e76_b1db_2faabf89ab3e.slice/crio-20f5f32739f5a06f87b523729cd0eede9851c5ff7b376b76fc3862032715a1be WatchSource:0}: Error finding container 20f5f32739f5a06f87b523729cd0eede9851c5ff7b376b76fc3862032715a1be: Status 404 returned error can't find the container with id 20f5f32739f5a06f87b523729cd0eede9851c5ff7b376b76fc3862032715a1be
Sep 09 16:33:14.119124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:14.118677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" event={"ID":"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e","Type":"ContainerStarted","Data":"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"}
Sep 09 16:33:14.119124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:14.118731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" event={"ID":"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e","Type":"ContainerStarted","Data":"20f5f32739f5a06f87b523729cd0eede9851c5ff7b376b76fc3862032715a1be"}
Sep 09 16:33:14.135832 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:14.135741    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" podStartSLOduration=1.135717959 podStartE2EDuration="1.135717959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:14.133520171 +0000 UTC m=+1174.083101019" watchObservedRunningTime="2026-09-09 16:33:14.135717959 +0000 UTC m=+1174.085298808"
Sep 09 16:33:15.124547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:15.124511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" event={"ID":"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b","Type":"ContainerStarted","Data":"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"}
Sep 09 16:33:15.124916 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:15.124732    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:15.140951 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:15.140905    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" podStartSLOduration=2.444117417 podStartE2EDuration="4.140890752s" totalImagesPullingTime="1.696773335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:15.139165198 +0000 UTC m=+1175.088746046" watchObservedRunningTime="2026-09-09 16:33:15.140890752 +0000 UTC m=+1175.090471647"
Sep 09 16:33:21.135720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:21.135690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:23.699364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:23.699326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:23.699720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:23.699378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:23.711622 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:23.711591    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:24.120707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:24.120672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:24.120904 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:24.120886    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:24.184308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:24.184281    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:33:41.412358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:41.412324    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:33:41.413886 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:41.413859    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:33:42.388290 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.388256    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:42.388527 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.388499    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" podUID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" containerName="maas-api" containerID="cri-o://d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c" gracePeriod=30
Sep 09 16:33:42.439260 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.439227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:42.439669 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.439443    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" podUID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" containerName="payload-processing" containerID="cri-o://767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf" gracePeriod=30
Sep 09 16:33:42.445846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.445809    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:42.446074 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.446033    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" podUID="4fa859cc-4b86-4170-bab6-4a78f778931f" containerName="payload-pre-processing" containerID="cri-o://515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b" gracePeriod=30
Sep 09 16:33:42.714382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.714308    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:42.732713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.732657    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:42.752895 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.752865    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:42.793832 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.793791    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume\") pod \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") "
Sep 09 16:33:42.794050 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.793899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz\") pod \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\" (UID: \"130d2291-27af-4ac2-bbd0-8f2483d4e22d\") "
Sep 09 16:33:42.794278 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.794253    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "130d2291-27af-4ac2-bbd0-8f2483d4e22d" (UID: "130d2291-27af-4ac2-bbd0-8f2483d4e22d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:42.795968 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.795934    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz" (OuterVolumeSpecName: "kube-api-access-tf9lz") pod "130d2291-27af-4ac2-bbd0-8f2483d4e22d" (UID: "130d2291-27af-4ac2-bbd0-8f2483d4e22d"). InnerVolumeSpecName "kube-api-access-tf9lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:42.895242 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895199    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume\") pod \"4fa859cc-4b86-4170-bab6-4a78f778931f\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") "
Sep 09 16:33:42.895242 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895245    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tgdh\" (UniqueName: \"kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh\") pod \"4fa859cc-4b86-4170-bab6-4a78f778931f\" (UID: \"4fa859cc-4b86-4170-bab6-4a78f778931f\") "
Sep 09 16:33:42.895505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895303    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") pod \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") "
Sep 09 16:33:42.895505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895371    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs\") pod \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") "
Sep 09 16:33:42.895505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895406    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b2x5\" (UniqueName: \"kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5\") pod \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") "
Sep 09 16:33:42.895505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895463    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca\") pod \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\" (UID: \"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b\") "
Sep 09 16:33:42.895881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895716    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fa859cc-4b86-4170-bab6-4a78f778931f" (UID: "4fa859cc-4b86-4170-bab6-4a78f778931f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:42.895881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895727    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9lz\" (UniqueName: \"kubernetes.io/projected/130d2291-27af-4ac2-bbd0-8f2483d4e22d-kube-api-access-tf9lz\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.895881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895872    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130d2291-27af-4ac2-bbd0-8f2483d4e22d-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.896146 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.895905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" (UID: "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:33:42.897621 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.897585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh" (OuterVolumeSpecName: "kube-api-access-5tgdh") pod "4fa859cc-4b86-4170-bab6-4a78f778931f" (UID: "4fa859cc-4b86-4170-bab6-4a78f778931f"). InnerVolumeSpecName "kube-api-access-5tgdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:42.897764 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.897698    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" (UID: "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:42.897764 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.897719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" (UID: "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:33:42.897764 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.897722    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5" (OuterVolumeSpecName: "kube-api-access-2b2x5") pod "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" (UID: "8b2b12e1-1f4f-443c-8273-7a3cbb99d04b"). InnerVolumeSpecName "kube-api-access-2b2x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:33:42.996975 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.996936    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa859cc-4b86-4170-bab6-4a78f778931f-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.997147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.996993    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgdh\" (UniqueName: \"kubernetes.io/projected/4fa859cc-4b86-4170-bab6-4a78f778931f-kube-api-access-5tgdh\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.997147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.997005    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.997147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.997016    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.997147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.997025    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2x5\" (UniqueName: \"kubernetes.io/projected/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-kube-api-access-2b2x5\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:42.997147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:42.997033    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:33:43.266946 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.266854    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa859cc-4b86-4170-bab6-4a78f778931f" containerID="515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b" exitCode=0
Sep 09 16:33:43.266946 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.266907    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"
Sep 09 16:33:43.266946 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.266939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" event={"ID":"4fa859cc-4b86-4170-bab6-4a78f778931f","Type":"ContainerDied","Data":"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"}
Sep 09 16:33:43.267253 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.267014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z" event={"ID":"4fa859cc-4b86-4170-bab6-4a78f778931f","Type":"ContainerDied","Data":"3285c8f9e87f97f6130b366551318167b398983394a86ad21edd63efb7769bfd"}
Sep 09 16:33:43.267253 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.267041    2559 scope.go:117] "RemoveContainer" containerID="515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"
Sep 09 16:33:43.268395 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.268372    2559 generic.go:356] "Generic (PLEG): container finished" podID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" containerID="767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf" exitCode=0
Sep 09 16:33:43.268510 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.268432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"
Sep 09 16:33:43.268510 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.268441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" event={"ID":"130d2291-27af-4ac2-bbd0-8f2483d4e22d","Type":"ContainerDied","Data":"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"}
Sep 09 16:33:43.268510 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.268491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd" event={"ID":"130d2291-27af-4ac2-bbd0-8f2483d4e22d","Type":"ContainerDied","Data":"32645d697d23d62ad83abc0a032e41a877f4a82c4d95ca7c7d1adc815d1f70e5"}
Sep 09 16:33:43.270094 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.269899    2559 generic.go:356] "Generic (PLEG): container finished" podID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" containerID="d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c" exitCode=0
Sep 09 16:33:43.270094 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.270005    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"
Sep 09 16:33:43.270094 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.270015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" event={"ID":"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b","Type":"ContainerDied","Data":"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"}
Sep 09 16:33:43.270094 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.270061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh" event={"ID":"8b2b12e1-1f4f-443c-8273-7a3cbb99d04b","Type":"ContainerDied","Data":"d343dcff520ae2860adbdeb178e1140018e4bdccecf914d89c08d3519f80e929"}
Sep 09 16:33:43.278562 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.278536    2559 scope.go:117] "RemoveContainer" containerID="515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"
Sep 09 16:33:43.278918 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:43.278898    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b\": container with ID starting with 515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b not found: ID does not exist" containerID="515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"
Sep 09 16:33:43.279002 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.278927    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b"} err="failed to get container status \"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b\": rpc error: code = NotFound desc = could not find container \"515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b\": container with ID starting with 515234c81d341335758030f7aa158388cfdd05142bde880cc27d596dea7c880b not found: ID does not exist"
Sep 09 16:33:43.279002 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.278946    2559 scope.go:117] "RemoveContainer" containerID="767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"
Sep 09 16:33:43.287991 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.287971    2559 scope.go:117] "RemoveContainer" containerID="767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"
Sep 09 16:33:43.288327 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:43.288308    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf\": container with ID starting with 767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf not found: ID does not exist" containerID="767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"
Sep 09 16:33:43.288393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.288338    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf"} err="failed to get container status \"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf\": rpc error: code = NotFound desc = could not find container \"767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf\": container with ID starting with 767e64b8948a5f9beeb11db5ad772bf3b916b93531d845a83b33db8414a37eaf not found: ID does not exist"
Sep 09 16:33:43.288393 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.288364    2559 scope.go:117] "RemoveContainer" containerID="d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"
Sep 09 16:33:43.299316 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.299269    2559 scope.go:117] "RemoveContainer" containerID="d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"
Sep 09 16:33:43.299799 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:33:43.299777    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c\": container with ID starting with d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c not found: ID does not exist" containerID="d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"
Sep 09 16:33:43.299858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.299807    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c"} err="failed to get container status \"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c\": rpc error: code = NotFound desc = could not find container \"d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c\": container with ID starting with d974044ec21ffa65f44c1f0f80d252ab5d19ebc89278b6d22e3e2b5e399ec23c not found: ID does not exist"
Sep 09 16:33:43.319340 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.319293    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:43.337535 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.337502    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9e0f78a3-57558cb67d-2zbhd"]
Sep 09 16:33:43.354608 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.354573    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:43.360414 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.360382    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e0f78a3-ddbd97d69-2z7nh"]
Sep 09 16:33:43.370132 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.370102    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:43.374027 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:43.373994    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9e0f78a3-6db54bbf5d-tdl2z"]
Sep 09 16:33:44.531646 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:44.531616    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" path="/var/lib/kubelet/pods/130d2291-27af-4ac2-bbd0-8f2483d4e22d/volumes"
Sep 09 16:33:44.532056 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:44.532003    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa859cc-4b86-4170-bab6-4a78f778931f" path="/var/lib/kubelet/pods/4fa859cc-4b86-4170-bab6-4a78f778931f/volumes"
Sep 09 16:33:44.532371 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:44.532357    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" path="/var/lib/kubelet/pods/8b2b12e1-1f4f-443c-8273-7a3cbb99d04b/volumes"
Sep 09 16:33:48.837901 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.837863    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838232    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" containerName="maas-api"
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838243    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" containerName="maas-api"
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838252    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" containerName="payload-processing"
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838258    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" containerName="payload-processing"
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838276    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa859cc-4b86-4170-bab6-4a78f778931f" containerName="payload-pre-processing"
Sep 09 16:33:48.838330 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838282    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa859cc-4b86-4170-bab6-4a78f778931f" containerName="payload-pre-processing"
Sep 09 16:33:48.838540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838336    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="130d2291-27af-4ac2-bbd0-8f2483d4e22d" containerName="payload-processing"
Sep 09 16:33:48.838540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838349    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa859cc-4b86-4170-bab6-4a78f778931f" containerName="payload-pre-processing"
Sep 09 16:33:48.838540 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.838355    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b2b12e1-1f4f-443c-8273-7a3cbb99d04b" containerName="maas-api"
Sep 09 16:33:48.843889 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.843869    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:48.846481 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.846461    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-49eb9be1\""
Sep 09 16:33:48.846590 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.846515    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-49eb9be1-dockercfg-j679f\""
Sep 09 16:33:48.851715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.851691    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:33:48.877830 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.877804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:33:48.881757 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.881741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:48.888796 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.888701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:33:48.954713 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.954680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgh92\" (UniqueName: \"kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:48.954863 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:48.954763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.055791 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.055728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgh92\" (UniqueName: \"kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.056020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.055808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd4n\" (UniqueName: \"kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.056020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.055884    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.056020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.055932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.056541 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.056519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.064170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.064141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgh92\" (UniqueName: \"kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92\") pod \"payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.155214 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.155137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:49.156415 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.156390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.156536 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.156444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd4n\" (UniqueName: \"kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.157098 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.157077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.165456 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.165436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd4n\" (UniqueName: \"kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n\") pod \"payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") " pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.192375 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.192343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:49.427113 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.427026    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:33:49.429893 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:49.429860    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064a0ec6_e133_4060_8a9d_012b7d6593f0.slice/crio-0673269790ccf63411c0943afacbb4d01a8ced158e1a76a7657832919709c94c WatchSource:0}: Error finding container 0673269790ccf63411c0943afacbb4d01a8ced158e1a76a7657832919709c94c: Status 404 returned error can't find the container with id 0673269790ccf63411c0943afacbb4d01a8ced158e1a76a7657832919709c94c
Sep 09 16:33:49.479635 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:49.479609    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:33:49.486988 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:33:49.486939    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2380cf_b3e3_4c53_93ed_fce38226aa4c.slice/crio-9edd4d09d74bc79575c59f6e2041d0ac4f1d1266e0a62c97d74cb5472c2a04de WatchSource:0}: Error finding container 9edd4d09d74bc79575c59f6e2041d0ac4f1d1266e0a62c97d74cb5472c2a04de: Status 404 returned error can't find the container with id 9edd4d09d74bc79575c59f6e2041d0ac4f1d1266e0a62c97d74cb5472c2a04de
Sep 09 16:33:50.309364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.309325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" event={"ID":"0a2380cf-b3e3-4c53-93ed-fce38226aa4c","Type":"ContainerStarted","Data":"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"}
Sep 09 16:33:50.309871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.309399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" event={"ID":"0a2380cf-b3e3-4c53-93ed-fce38226aa4c","Type":"ContainerStarted","Data":"9edd4d09d74bc79575c59f6e2041d0ac4f1d1266e0a62c97d74cb5472c2a04de"}
Sep 09 16:33:50.309871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.309462    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:33:50.310941 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.310919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" event={"ID":"064a0ec6-e133-4060-8a9d-012b7d6593f0","Type":"ContainerStarted","Data":"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"}
Sep 09 16:33:50.311048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.310955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" event={"ID":"064a0ec6-e133-4060-8a9d-012b7d6593f0","Type":"ContainerStarted","Data":"0673269790ccf63411c0943afacbb4d01a8ced158e1a76a7657832919709c94c"}
Sep 09 16:33:50.311048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.311013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:33:50.324792 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.324751    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" podStartSLOduration=2.324740476 podStartE2EDuration="2.324740476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:50.323284623 +0000 UTC m=+1210.272865471" watchObservedRunningTime="2026-09-09 16:33:50.324740476 +0000 UTC m=+1210.274321323"
Sep 09 16:33:50.338529 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:33:50.338485    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" podStartSLOduration=2.33847344 podStartE2EDuration="2.33847344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:33:50.336927021 +0000 UTC m=+1210.286507872" watchObservedRunningTime="2026-09-09 16:33:50.33847344 +0000 UTC m=+1210.288054288"
Sep 09 16:34:01.317799 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:01.317766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:34:01.320185 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:01.317836    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:34:17.687742 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.687705    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:34:17.688740 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.687923    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" podUID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" containerName="payload-processing" containerID="cri-o://14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767" gracePeriod=30
Sep 09 16:34:17.695663 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.695639    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:34:17.695835 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.695812    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" podUID="064a0ec6-e133-4060-8a9d-012b7d6593f0" containerName="payload-pre-processing" containerID="cri-o://5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746" gracePeriod=30
Sep 09 16:34:17.944844 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.944796    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:34:17.967171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.967149    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:34:17.992499 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.992477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume\") pod \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") "
Sep 09 16:34:17.992616 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.992524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvd4n\" (UniqueName: \"kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n\") pod \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\" (UID: \"0a2380cf-b3e3-4c53-93ed-fce38226aa4c\") "
Sep 09 16:34:17.992848 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.992822    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a2380cf-b3e3-4c53-93ed-fce38226aa4c" (UID: "0a2380cf-b3e3-4c53-93ed-fce38226aa4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:17.994570 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:17.994550    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n" (OuterVolumeSpecName: "kube-api-access-cvd4n") pod "0a2380cf-b3e3-4c53-93ed-fce38226aa4c" (UID: "0a2380cf-b3e3-4c53-93ed-fce38226aa4c"). InnerVolumeSpecName "kube-api-access-cvd4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:18.093059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.093038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgh92\" (UniqueName: \"kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92\") pod \"064a0ec6-e133-4060-8a9d-012b7d6593f0\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") "
Sep 09 16:34:18.093176 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.093084    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume\") pod \"064a0ec6-e133-4060-8a9d-012b7d6593f0\" (UID: \"064a0ec6-e133-4060-8a9d-012b7d6593f0\") "
Sep 09 16:34:18.093349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.093332    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.093420 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.093353    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvd4n\" (UniqueName: \"kubernetes.io/projected/0a2380cf-b3e3-4c53-93ed-fce38226aa4c-kube-api-access-cvd4n\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.093503 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.093482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "064a0ec6-e133-4060-8a9d-012b7d6593f0" (UID: "064a0ec6-e133-4060-8a9d-012b7d6593f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:18.095154 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.095134    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92" (OuterVolumeSpecName: "kube-api-access-pgh92") pod "064a0ec6-e133-4060-8a9d-012b7d6593f0" (UID: "064a0ec6-e133-4060-8a9d-012b7d6593f0"). InnerVolumeSpecName "kube-api-access-pgh92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:18.193730 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.193705    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/064a0ec6-e133-4060-8a9d-012b7d6593f0-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.193730 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.193725    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgh92\" (UniqueName: \"kubernetes.io/projected/064a0ec6-e133-4060-8a9d-012b7d6593f0-kube-api-access-pgh92\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:18.453684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.453648    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" containerID="14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767" exitCode=0
Sep 09 16:34:18.453846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.453710    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"
Sep 09 16:34:18.453846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.453730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" event={"ID":"0a2380cf-b3e3-4c53-93ed-fce38226aa4c","Type":"ContainerDied","Data":"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"}
Sep 09 16:34:18.453846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.453814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx" event={"ID":"0a2380cf-b3e3-4c53-93ed-fce38226aa4c","Type":"ContainerDied","Data":"9edd4d09d74bc79575c59f6e2041d0ac4f1d1266e0a62c97d74cb5472c2a04de"}
Sep 09 16:34:18.453846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.453844    2559 scope.go:117] "RemoveContainer" containerID="14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"
Sep 09 16:34:18.455104 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.455078    2559 generic.go:356] "Generic (PLEG): container finished" podID="064a0ec6-e133-4060-8a9d-012b7d6593f0" containerID="5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746" exitCode=0
Sep 09 16:34:18.455216 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.455128    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"
Sep 09 16:34:18.455216 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.455163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" event={"ID":"064a0ec6-e133-4060-8a9d-012b7d6593f0","Type":"ContainerDied","Data":"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"}
Sep 09 16:34:18.455331 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.455221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2" event={"ID":"064a0ec6-e133-4060-8a9d-012b7d6593f0","Type":"ContainerDied","Data":"0673269790ccf63411c0943afacbb4d01a8ced158e1a76a7657832919709c94c"}
Sep 09 16:34:18.462941 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.462923    2559 scope.go:117] "RemoveContainer" containerID="14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"
Sep 09 16:34:18.463261 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:18.463227    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767\": container with ID starting with 14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767 not found: ID does not exist" containerID="14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"
Sep 09 16:34:18.463375 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.463268    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767"} err="failed to get container status \"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767\": rpc error: code = NotFound desc = could not find container \"14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767\": container with ID starting with 14d2f718b47fef8235619244a2c907a821ddbaf5fea473c6217116c070072767 not found: ID does not exist"
Sep 09 16:34:18.463375 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.463291    2559 scope.go:117] "RemoveContainer" containerID="5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"
Sep 09 16:34:18.471049 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.471033    2559 scope.go:117] "RemoveContainer" containerID="5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"
Sep 09 16:34:18.471328 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:18.471310    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746\": container with ID starting with 5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746 not found: ID does not exist" containerID="5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"
Sep 09 16:34:18.471388 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.471334    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746"} err="failed to get container status \"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746\": rpc error: code = NotFound desc = could not find container \"5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746\": container with ID starting with 5625fae6bdcb86a37a65f2be795d251403c34e8498ab5fec159f8d253352f746 not found: ID does not exist"
Sep 09 16:34:18.476701 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.476678    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:34:18.481042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.481023    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49eb9be1-7d99fd7578-d4gj2"]
Sep 09 16:34:18.490929 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.490907    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:34:18.495405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.495385    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-49eb9be1-7bcbb96d8f-wvtqx"]
Sep 09 16:34:18.532069 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.532040    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="064a0ec6-e133-4060-8a9d-012b7d6593f0" path="/var/lib/kubelet/pods/064a0ec6-e133-4060-8a9d-012b7d6593f0/volumes"
Sep 09 16:34:18.532392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:18.532379    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" path="/var/lib/kubelet/pods/0a2380cf-b3e3-4c53-93ed-fce38226aa4c/volumes"
Sep 09 16:34:24.039548 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.039513    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:24.040147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040121    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="064a0ec6-e133-4060-8a9d-012b7d6593f0" containerName="payload-pre-processing"
Sep 09 16:34:24.040147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040147    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="064a0ec6-e133-4060-8a9d-012b7d6593f0" containerName="payload-pre-processing"
Sep 09 16:34:24.040296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" containerName="payload-processing"
Sep 09 16:34:24.040296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040171    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" containerName="payload-processing"
Sep 09 16:34:24.040296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040290    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="064a0ec6-e133-4060-8a9d-012b7d6593f0" containerName="payload-pre-processing"
Sep 09 16:34:24.040390 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.040304    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2380cf-b3e3-4c53-93ed-fce38226aa4c" containerName="payload-processing"
Sep 09 16:34:24.045068 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.045051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.047400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.047379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-20ed533b-serving-cert\""
Sep 09 16:34:24.048547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.048520    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k96v8\""
Sep 09 16:34:24.048646 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.048581    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 16:34:24.048646 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.048593    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 16:34:24.048646 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.048602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:34:24.051042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.051022    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:24.142265 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.142237    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnv2\" (UniqueName: \"kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.142385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.142271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.142385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.142337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.142385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.142352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.243620 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.243598    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.243756 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.243625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.243756 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.243679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnv2\" (UniqueName: \"kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.243756 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.243700    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.243877 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:24.243809    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-serving-cert: secret "maas-api-e2e-derive-20ed533b-serving-cert" not found
Sep 09 16:34:24.243877 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:24.243872    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls podName:7ab2f176-d5a4-4091-b3f3-40346e030f3c nodeName:}" failed. No retries permitted until 2026-09-09 16:34:24.743852765 +0000 UTC m=+1244.693433605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls") pod "maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" (UID: "7ab2f176-d5a4-4091-b3f3-40346e030f3c") : secret "maas-api-e2e-derive-20ed533b-serving-cert" not found
Sep 09 16:34:24.244335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.244311    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.246115 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.246092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.254051 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.254028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnv2\" (UniqueName: \"kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.639442 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.639406    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:24.642832 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.642811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.645127 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.645108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-20ed533b\""
Sep 09 16:34:24.645127 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.645121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-20ed533b-dockercfg-ssx6b\""
Sep 09 16:34:24.651708 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.651672    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:24.748135 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.748099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9fr\" (UniqueName: \"kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.748286 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.748162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.748286 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.748230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.750449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.750422    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") pod \"maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:24.848637 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.848609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9fr\" (UniqueName: \"kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.848733 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.848701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.849376 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.849358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.856162 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.856140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9fr\" (UniqueName: \"kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr\") pod \"payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") " pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.955366 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.955300    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:24.959057 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:24.959041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:25.226945 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.226758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:25.230368 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:34:25.230338    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcdbb7d_6c07_4ecd_ba07_d094f444932c.slice/crio-347d3d699aef2ec54ea255000da250926da5f2bb6c0eb32caeb6a431e04eac31 WatchSource:0}: Error finding container 347d3d699aef2ec54ea255000da250926da5f2bb6c0eb32caeb6a431e04eac31: Status 404 returned error can't find the container with id 347d3d699aef2ec54ea255000da250926da5f2bb6c0eb32caeb6a431e04eac31
Sep 09 16:34:25.231204 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.231184    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:25.233556 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:34:25.233537    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab2f176_d5a4_4091_b3f3_40346e030f3c.slice/crio-0b7fb76882a11122635f18a560a66f7e672a43a9b0990f928a80a1f75ed344e8 WatchSource:0}: Error finding container 0b7fb76882a11122635f18a560a66f7e672a43a9b0990f928a80a1f75ed344e8: Status 404 returned error can't find the container with id 0b7fb76882a11122635f18a560a66f7e672a43a9b0990f928a80a1f75ed344e8
Sep 09 16:34:25.494978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.494875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" event={"ID":"7fcdbb7d-6c07-4ecd-ba07-d094f444932c","Type":"ContainerStarted","Data":"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"}
Sep 09 16:34:25.494978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.494933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" event={"ID":"7fcdbb7d-6c07-4ecd-ba07-d094f444932c","Type":"ContainerStarted","Data":"347d3d699aef2ec54ea255000da250926da5f2bb6c0eb32caeb6a431e04eac31"}
Sep 09 16:34:25.495182 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.494984    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:25.496071 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.496047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" event={"ID":"7ab2f176-d5a4-4091-b3f3-40346e030f3c","Type":"ContainerStarted","Data":"0b7fb76882a11122635f18a560a66f7e672a43a9b0990f928a80a1f75ed344e8"}
Sep 09 16:34:25.509608 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:25.509565    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" podStartSLOduration=1.5095511350000002 podStartE2EDuration="1.509551135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:25.508136609 +0000 UTC m=+1245.457717455" watchObservedRunningTime="2026-09-09 16:34:25.509551135 +0000 UTC m=+1245.459131983"
Sep 09 16:34:26.504179 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:26.504085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" event={"ID":"7ab2f176-d5a4-4091-b3f3-40346e030f3c","Type":"ContainerStarted","Data":"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"}
Sep 09 16:34:26.527978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:26.524094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" podStartSLOduration=1.716035362 podStartE2EDuration="2.524073009s" totalImagesPullingTime="808.037647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:26.521126755 +0000 UTC m=+1246.470707603" watchObservedRunningTime="2026-09-09 16:34:26.524073009 +0000 UTC m=+1246.473653858"
Sep 09 16:34:27.509707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:27.509680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:33.521160 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:33.521079    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:36.506940 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:36.506907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:53.844374 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:53.844329    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:53.844989 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:53.844587    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" podUID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" containerName="maas-api" containerID="cri-o://ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f" gracePeriod=30
Sep 09 16:34:53.901128 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:53.901091    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:53.901365 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:53.901338    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" podUID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" containerName="payload-processing" containerID="cri-o://3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361" gracePeriod=30
Sep 09 16:34:54.153862 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.153837    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:54.196389 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.196360    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:54.313572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313536    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") pod \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") "
Sep 09 16:34:54.313572 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313576    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca\") pod \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") "
Sep 09 16:34:54.313813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdnv2\" (UniqueName: \"kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2\") pod \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") "
Sep 09 16:34:54.313813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313624    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs\") pod \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\" (UID: \"7ab2f176-d5a4-4091-b3f3-40346e030f3c\") "
Sep 09 16:34:54.313813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313659    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w9fr\" (UniqueName: \"kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr\") pod \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") "
Sep 09 16:34:54.313813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume\") pod \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\" (UID: \"7fcdbb7d-6c07-4ecd-ba07-d094f444932c\") "
Sep 09 16:34:54.314037 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.313983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ab2f176-d5a4-4091-b3f3-40346e030f3c" (UID: "7ab2f176-d5a4-4091-b3f3-40346e030f3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:54.314375 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.314348    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fcdbb7d-6c07-4ecd-ba07-d094f444932c" (UID: "7fcdbb7d-6c07-4ecd-ba07-d094f444932c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:34:54.315918 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.315891    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2" (OuterVolumeSpecName: "kube-api-access-cdnv2") pod "7ab2f176-d5a4-4091-b3f3-40346e030f3c" (UID: "7ab2f176-d5a4-4091-b3f3-40346e030f3c"). InnerVolumeSpecName "kube-api-access-cdnv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:54.315918 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.315896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ab2f176-d5a4-4091-b3f3-40346e030f3c" (UID: "7ab2f176-d5a4-4091-b3f3-40346e030f3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:34:54.316183 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.316157    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ab2f176-d5a4-4091-b3f3-40346e030f3c" (UID: "7ab2f176-d5a4-4091-b3f3-40346e030f3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:34:54.316183 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.316158    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr" (OuterVolumeSpecName: "kube-api-access-8w9fr") pod "7fcdbb7d-6c07-4ecd-ba07-d094f444932c" (UID: "7fcdbb7d-6c07-4ecd-ba07-d094f444932c"). InnerVolumeSpecName "kube-api-access-8w9fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414744    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414780    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab2f176-d5a4-4091-b3f3-40346e030f3c-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414794    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdnv2\" (UniqueName: \"kubernetes.io/projected/7ab2f176-d5a4-4091-b3f3-40346e030f3c-kube-api-access-cdnv2\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414805    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab2f176-d5a4-4091-b3f3-40346e030f3c-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414816    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9fr\" (UniqueName: \"kubernetes.io/projected/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-kube-api-access-8w9fr\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.414846 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.414827    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fcdbb7d-6c07-4ecd-ba07-d094f444932c-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:34:54.644603 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.644567    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" containerID="ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f" exitCode=0
Sep 09 16:34:54.644779 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.644632    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"
Sep 09 16:34:54.644779 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.644658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" event={"ID":"7ab2f176-d5a4-4091-b3f3-40346e030f3c","Type":"ContainerDied","Data":"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"}
Sep 09 16:34:54.644779 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.644711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz" event={"ID":"7ab2f176-d5a4-4091-b3f3-40346e030f3c","Type":"ContainerDied","Data":"0b7fb76882a11122635f18a560a66f7e672a43a9b0990f928a80a1f75ed344e8"}
Sep 09 16:34:54.644779 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.644736    2559 scope.go:117] "RemoveContainer" containerID="ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"
Sep 09 16:34:54.646075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.646053    2559 generic.go:356] "Generic (PLEG): container finished" podID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" containerID="3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361" exitCode=0
Sep 09 16:34:54.646159 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.646084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" event={"ID":"7fcdbb7d-6c07-4ecd-ba07-d094f444932c","Type":"ContainerDied","Data":"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"}
Sep 09 16:34:54.646159 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.646110    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"
Sep 09 16:34:54.646159 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.646139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs" event={"ID":"7fcdbb7d-6c07-4ecd-ba07-d094f444932c","Type":"ContainerDied","Data":"347d3d699aef2ec54ea255000da250926da5f2bb6c0eb32caeb6a431e04eac31"}
Sep 09 16:34:54.653855 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.653832    2559 scope.go:117] "RemoveContainer" containerID="ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"
Sep 09 16:34:54.654217 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:54.654198    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f\": container with ID starting with ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f not found: ID does not exist" containerID="ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"
Sep 09 16:34:54.654294 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.654229    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f"} err="failed to get container status \"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f\": rpc error: code = NotFound desc = could not find container \"ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f\": container with ID starting with ee420c30ce40646530d29088b51eae2f72fee38fec772300a5dd7bd3d1e7fd0f not found: ID does not exist"
Sep 09 16:34:54.654294 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.654245    2559 scope.go:117] "RemoveContainer" containerID="3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"
Sep 09 16:34:54.662226 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.661766    2559 scope.go:117] "RemoveContainer" containerID="3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"
Sep 09 16:34:54.662226 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:34:54.662075    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361\": container with ID starting with 3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361 not found: ID does not exist" containerID="3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"
Sep 09 16:34:54.662226 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.662104    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361"} err="failed to get container status \"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361\": rpc error: code = NotFound desc = could not find container \"3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361\": container with ID starting with 3d7540687fcc742241f40e49cea879a5a0f4aca4c5b8eff77addcbff405bb361 not found: ID does not exist"
Sep 09 16:34:54.664102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.664085    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:54.667563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.667516    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-20ed533b-856f4fddbd-zx8fz"]
Sep 09 16:34:54.677065 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.677034    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:54.678939 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:54.678919    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-20ed533b-5944c8d6b5-9pmxs"]
Sep 09 16:34:56.535866 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:56.535824    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" path="/var/lib/kubelet/pods/7ab2f176-d5a4-4091-b3f3-40346e030f3c/volumes"
Sep 09 16:34:56.536362 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:56.536322    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" path="/var/lib/kubelet/pods/7fcdbb7d-6c07-4ecd-ba07-d094f444932c/volumes"
Sep 09 16:34:58.500703 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.500666    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:34:58.501185 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501169    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" containerName="payload-processing"
Sep 09 16:34:58.501232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501187    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" containerName="payload-processing"
Sep 09 16:34:58.501232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501208    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" containerName="maas-api"
Sep 09 16:34:58.501232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501214    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" containerName="maas-api"
Sep 09 16:34:58.501335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501279    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab2f176-d5a4-4091-b3f3-40346e030f3c" containerName="maas-api"
Sep 09 16:34:58.501335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.501289    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcdbb7d-6c07-4ecd-ba07-d094f444932c" containerName="payload-processing"
Sep 09 16:34:58.505921 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.505895    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.509171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.509130    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-630acf\"/\"default-dockercfg-lczkm\""
Sep 09 16:34:58.509171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.509167    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-630acf\"/\"test-model-630acf-kserve-self-signed-certs\""
Sep 09 16:34:58.509378 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.509188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-630acf\"/\"kube-root-ca.crt\""
Sep 09 16:34:58.509378 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.509136    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-630acf\"/\"openshift-service-ca.crt\""
Sep 09 16:34:58.510649 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.510623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:34:58.647237 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.647451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.647451 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.647570 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.647570 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.647570 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.647560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jls\" (UniqueName: \"kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jls\" (UniqueName: \"kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.748894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.749034 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.748948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.750800 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.750735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.751007 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.750990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.755449 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.755421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jls\" (UniqueName: \"kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls\") pod \"test-model-630acf-kserve-798cbd9475-6q22d\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") " pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:58.817162 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:58.817125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:34:59.088379 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:59.088353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:34:59.091098 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:34:59.091073    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b6b0115_5858_4d20_85c3_977c3b3e6821.slice/crio-aa71bf8d17b103fa164c6923366cf2166fc3e4adb42b9d20c42473c74115bf89 WatchSource:0}: Error finding container aa71bf8d17b103fa164c6923366cf2166fc3e4adb42b9d20c42473c74115bf89: Status 404 returned error can't find the container with id aa71bf8d17b103fa164c6923366cf2166fc3e4adb42b9d20c42473c74115bf89
Sep 09 16:34:59.676161 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:59.676125    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" event={"ID":"7b6b0115-5858-4d20-85c3-977c3b3e6821","Type":"ContainerStarted","Data":"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"}
Sep 09 16:34:59.676161 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:59.676164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" event={"ID":"7b6b0115-5858-4d20-85c3-977c3b3e6821","Type":"ContainerStarted","Data":"aa71bf8d17b103fa164c6923366cf2166fc3e4adb42b9d20c42473c74115bf89"}
Sep 09 16:34:59.691717 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:34:59.691659    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" podStartSLOduration=1.691641543 podStartE2EDuration="1.691641543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:34:59.691343939 +0000 UTC m=+1279.640924786" watchObservedRunningTime="2026-09-09 16:34:59.691641543 +0000 UTC m=+1279.641222426"
Sep 09 16:35:08.818275 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:08.818235    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:35:08.818669 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:08.818291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:35:08.830736 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:08.830703    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:35:09.737991 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:09.737952    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:35:15.690694 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.690662    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:15.693921 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.693905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.696153 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.696135    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-563e2ddd-kserve-self-signed-certs\""
Sep 09 16:35:15.700622 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.700596    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:15.800110 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.800110 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.800333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.800333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8ln\" (UniqueName: \"kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.800333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.800333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.800294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901166 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8ln\" (UniqueName: \"kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901606 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901656 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.901704 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.901686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.903563 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.903539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.903645 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.903631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:15.907863 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:15.907844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8ln\" (UniqueName: \"kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln\") pod \"e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") " pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:16.003805 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:16.003721    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:16.272899 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:16.272723    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:16.275605 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:35:16.275568    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb58500e_6105_4b73_bdbf_18bdd069bbba.slice/crio-12f4f1c84923ab8b351e84c55dbcdd22e03074b9a4fe787785ff54b2aaba4250 WatchSource:0}: Error finding container 12f4f1c84923ab8b351e84c55dbcdd22e03074b9a4fe787785ff54b2aaba4250: Status 404 returned error can't find the container with id 12f4f1c84923ab8b351e84c55dbcdd22e03074b9a4fe787785ff54b2aaba4250
Sep 09 16:35:16.763295 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:16.763258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" event={"ID":"cb58500e-6105-4b73-bdbf-18bdd069bbba","Type":"ContainerStarted","Data":"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"}
Sep 09 16:35:16.763295 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:16.763307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" event={"ID":"cb58500e-6105-4b73-bdbf-18bdd069bbba","Type":"ContainerStarted","Data":"12f4f1c84923ab8b351e84c55dbcdd22e03074b9a4fe787785ff54b2aaba4250"}
Sep 09 16:35:16.779070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:16.779015    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" podStartSLOduration=1.778998547 podStartE2EDuration="1.778998547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:35:16.776754499 +0000 UTC m=+1296.726335345" watchObservedRunningTime="2026-09-09 16:35:16.778998547 +0000 UTC m=+1296.728579392"
Sep 09 16:35:21.854813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:21.854777    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:35:21.855284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:21.855066    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" podUID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" containerName="payload-pre-processing" containerID="cri-o://75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc" gracePeriod=30
Sep 09 16:35:22.103196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.103162    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:35:22.156992 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.156877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume\") pod \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") "
Sep 09 16:35:22.157132 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.157040    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lcjt\" (UniqueName: \"kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt\") pod \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\" (UID: \"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44\") "
Sep 09 16:35:22.157368 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.157339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" (UID: "6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:35:22.159171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.159141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt" (OuterVolumeSpecName: "kube-api-access-5lcjt") pod "6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" (UID: "6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44"). InnerVolumeSpecName "kube-api-access-5lcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:22.258013 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.257945    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lcjt\" (UniqueName: \"kubernetes.io/projected/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-kube-api-access-5lcjt\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:22.258013 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.258011    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:22.796721 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.796628    2559 generic.go:356] "Generic (PLEG): container finished" podID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" containerID="75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc" exitCode=0
Sep 09 16:35:22.796721 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.796682    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"
Sep 09 16:35:22.796721 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.796704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" event={"ID":"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44","Type":"ContainerDied","Data":"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"}
Sep 09 16:35:22.796980 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.796757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g" event={"ID":"6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44","Type":"ContainerDied","Data":"ec4c4f3e01168b1f6e7c54e8449be11e134eaa5e07edb22efcef4fe1280f565f"}
Sep 09 16:35:22.796980 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.796779    2559 scope.go:117] "RemoveContainer" containerID="75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"
Sep 09 16:35:22.805564 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.805543    2559 scope.go:117] "RemoveContainer" containerID="75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"
Sep 09 16:35:22.805889 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:35:22.805865    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc\": container with ID starting with 75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc not found: ID does not exist" containerID="75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"
Sep 09 16:35:22.805982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.805899    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc"} err="failed to get container status \"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc\": rpc error: code = NotFound desc = could not find container \"75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc\": container with ID starting with 75fcafaf83539b24b1288e6a396e7482ba76803bcb45a7e2e1dd34cb155db7dc not found: ID does not exist"
Sep 09 16:35:22.812529 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.812496    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:35:22.818033 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:22.818002    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-05e496-559c465594-6cf4g"]
Sep 09 16:35:24.533204 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:24.533165    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" path="/var/lib/kubelet/pods/6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44/volumes"
Sep 09 16:35:26.004651 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:26.004607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:26.005141 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:26.004669    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:26.016782 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:26.016750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:26.828778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:26.828749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:37.817284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:37.817242    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:39.285760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:39.285727    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:35:52.837443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:52.837376    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" podUID="cb58500e-6105-4b73-bdbf-18bdd069bbba" containerName="main" containerID="cri-o://13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428" gracePeriod=45
Sep 09 16:35:53.090021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.089942    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:53.135718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135678    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135726    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135811    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135915    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8ln\" (UniqueName: \"kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln\") pod \"cb58500e-6105-4b73-bdbf-18bdd069bbba\" (UID: \"cb58500e-6105-4b73-bdbf-18bdd069bbba\") "
Sep 09 16:35:53.135982 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.135946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:53.136254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.136131    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:53.136254 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.136148    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-tmp-dir\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.136358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.136278    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home" (OuterVolumeSpecName: "home") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:53.138245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.138210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm" (OuterVolumeSpecName: "dshm") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:53.138377 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.138320    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln" (OuterVolumeSpecName: "kube-api-access-jw8ln") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "kube-api-access-jw8ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:53.138377 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.138325    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb58500e-6105-4b73-bdbf-18bdd069bbba" (UID: "cb58500e-6105-4b73-bdbf-18bdd069bbba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:53.237008 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.236974    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8ln\" (UniqueName: \"kubernetes.io/projected/cb58500e-6105-4b73-bdbf-18bdd069bbba-kube-api-access-jw8ln\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.237008 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.237007    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-home\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.237207 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.237021    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb58500e-6105-4b73-bdbf-18bdd069bbba-tls-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.237207 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.237031    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-model-cache\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.237207 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.237040    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb58500e-6105-4b73-bdbf-18bdd069bbba-dshm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:53.949800 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.949766    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb58500e-6105-4b73-bdbf-18bdd069bbba" containerID="13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428" exitCode=0
Sep 09 16:35:53.950215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.949806    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"
Sep 09 16:35:53.950215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.949831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" event={"ID":"cb58500e-6105-4b73-bdbf-18bdd069bbba","Type":"ContainerDied","Data":"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"}
Sep 09 16:35:53.950215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.949868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p" event={"ID":"cb58500e-6105-4b73-bdbf-18bdd069bbba","Type":"ContainerDied","Data":"12f4f1c84923ab8b351e84c55dbcdd22e03074b9a4fe787785ff54b2aaba4250"}
Sep 09 16:35:53.950215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.949889    2559 scope.go:117] "RemoveContainer" containerID="13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"
Sep 09 16:35:53.959256 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.959240    2559 scope.go:117] "RemoveContainer" containerID="13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"
Sep 09 16:35:53.959547 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:35:53.959529    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428\": container with ID starting with 13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428 not found: ID does not exist" containerID="13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"
Sep 09 16:35:53.959590 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.959557    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428"} err="failed to get container status \"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428\": rpc error: code = NotFound desc = could not find container \"13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428\": container with ID starting with 13e91a5f31d9df0b0b159467755c1b0b65059d91539e5adbd63502574ec01428 not found: ID does not exist"
Sep 09 16:35:53.969922 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.969900    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:53.973919 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:53.973900    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-563e2ddd-kserve-67dc59586d-bl28p"]
Sep 09 16:35:54.305525 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.305473    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" podUID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" containerName="main" containerID="cri-o://63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13" gracePeriod=45
Sep 09 16:35:54.531224 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.531194    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb58500e-6105-4b73-bdbf-18bdd069bbba" path="/var/lib/kubelet/pods/cb58500e-6105-4b73-bdbf-18bdd069bbba/volumes"
Sep 09 16:35:54.556058 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.556007    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:35:54.649691 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.649847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649727    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtkbp\" (UniqueName: \"kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.649847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.649847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.650035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.650035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home\") pod \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\" (UID: \"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e\") "
Sep 09 16:35:54.650035 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.649982    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:54.650213 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.650193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:54.650281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.650227    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tmp-dir\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.650369 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.650332    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home" (OuterVolumeSpecName: "home") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:54.651841 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.651811    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:35:54.652027 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.651951    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm" (OuterVolumeSpecName: "dshm") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:35:54.652102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.652060    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp" (OuterVolumeSpecName: "kube-api-access-dtkbp") pod "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" (UID: "f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e"). InnerVolumeSpecName "kube-api-access-dtkbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:35:54.751285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.751255    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-dshm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.751285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.751283    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-model-cache\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.751443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.751296    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-home\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.751443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.751315    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtkbp\" (UniqueName: \"kubernetes.io/projected/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-kube-api-access-dtkbp\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.751443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.751327    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e-tls-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:35:54.955282 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.955203    2559 generic.go:356] "Generic (PLEG): container finished" podID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" containerID="63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13" exitCode=0
Sep 09 16:35:54.955282 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.955252    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"
Sep 09 16:35:54.955718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.955298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" event={"ID":"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e","Type":"ContainerDied","Data":"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"}
Sep 09 16:35:54.955718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.955339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48" event={"ID":"f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e","Type":"ContainerDied","Data":"20f5f32739f5a06f87b523729cd0eede9851c5ff7b376b76fc3862032715a1be"}
Sep 09 16:35:54.955718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.955359    2559 scope.go:117] "RemoveContainer" containerID="63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"
Sep 09 16:35:54.968751 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.968732    2559 scope.go:117] "RemoveContainer" containerID="63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"
Sep 09 16:35:54.969038 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:35:54.969018    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13\": container with ID starting with 63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13 not found: ID does not exist" containerID="63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"
Sep 09 16:35:54.969113 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.969049    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13"} err="failed to get container status \"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13\": rpc error: code = NotFound desc = could not find container \"63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13\": container with ID starting with 63f0e9de8f3d4f526c38ed65243fcfdb759d8a8e89ed7d0b798afce887fd9a13 not found: ID does not exist"
Sep 09 16:35:54.978399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.978370    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:35:54.982194 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:54.982174    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-563e2ddd-kserve-6d67d94bc4-vqj48"]
Sep 09 16:35:56.531383 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:35:56.531350    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" path="/var/lib/kubelet/pods/f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e/volumes"
Sep 09 16:36:10.445801 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.445716    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446075    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" containerName="main"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446086    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" containerName="main"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446113    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" containerName="payload-pre-processing"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446119    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" containerName="payload-pre-processing"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446124    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb58500e-6105-4b73-bdbf-18bdd069bbba" containerName="main"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb58500e-6105-4b73-bdbf-18bdd069bbba" containerName="main"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446186    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbb9cdf-099d-49b5-b9fc-c1c3edf09e44" containerName="payload-pre-processing"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446194    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3bd2c33-4e4f-4e76-b1db-2faabf89ab3e" containerName="main"
Sep 09 16:36:10.446270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.446200    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb58500e-6105-4b73-bdbf-18bdd069bbba" containerName="main"
Sep 09 16:36:10.450606 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.450586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.452907 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.452885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-26563f\""
Sep 09 16:36:10.453078 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.452884    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-26563f-dockercfg-mnk2k\""
Sep 09 16:36:10.459101 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.459075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:36:10.477365 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.477324    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:36:10.481014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.480992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.490434 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.490402    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:36:10.593492 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.593451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.593812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.593569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbj6b\" (UniqueName: \"kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.593812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.593632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6w9p\" (UniqueName: \"kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.593812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.593782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.695078 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbj6b\" (UniqueName: \"kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.695078 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6w9p\" (UniqueName: \"kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.695343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.695343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.695873 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.695873 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.695858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.703146 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.703114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6w9p\" (UniqueName: \"kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p\") pod \"payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.703274 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.703113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbj6b\" (UniqueName: \"kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b\") pod \"payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") " pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:10.761735 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.761696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:10.792850 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:10.792810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:11.037031 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:11.037006    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:36:11.039451 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:36:11.039428    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e2ed23_3f2e_40a5_917e_f3bc7a65f7df.slice/crio-7e1f6a42dddd2d0a5e25b62046a98ba655b4a5c18eb6cce27de20117d48379e7 WatchSource:0}: Error finding container 7e1f6a42dddd2d0a5e25b62046a98ba655b4a5c18eb6cce27de20117d48379e7: Status 404 returned error can't find the container with id 7e1f6a42dddd2d0a5e25b62046a98ba655b4a5c18eb6cce27de20117d48379e7
Sep 09 16:36:11.075155 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:11.074987    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:36:11.086132 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:36:11.086106    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5befa65f_3583_49ac_91e4_63bc51be35c7.slice/crio-aee521a88f3ad9ab814f7a7e6fb6d5de818773f631fdbc52a56f0df9973821a7 WatchSource:0}: Error finding container aee521a88f3ad9ab814f7a7e6fb6d5de818773f631fdbc52a56f0df9973821a7: Status 404 returned error can't find the container with id aee521a88f3ad9ab814f7a7e6fb6d5de818773f631fdbc52a56f0df9973821a7
Sep 09 16:36:12.043720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.043683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" event={"ID":"5befa65f-3583-49ac-91e4-63bc51be35c7","Type":"ContainerStarted","Data":"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"}
Sep 09 16:36:12.044198 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.043778    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:12.044198 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.043802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" event={"ID":"5befa65f-3583-49ac-91e4-63bc51be35c7","Type":"ContainerStarted","Data":"aee521a88f3ad9ab814f7a7e6fb6d5de818773f631fdbc52a56f0df9973821a7"}
Sep 09 16:36:12.045364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.045326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" event={"ID":"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df","Type":"ContainerStarted","Data":"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"}
Sep 09 16:36:12.045476 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.045379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" event={"ID":"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df","Type":"ContainerStarted","Data":"7e1f6a42dddd2d0a5e25b62046a98ba655b4a5c18eb6cce27de20117d48379e7"}
Sep 09 16:36:12.045476 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.045436    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:12.058276 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.058218    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" podStartSLOduration=2.05820048 podStartE2EDuration="2.05820048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:12.057311814 +0000 UTC m=+1352.006892660" watchObservedRunningTime="2026-09-09 16:36:12.05820048 +0000 UTC m=+1352.007781325"
Sep 09 16:36:12.072232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:12.072176    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" podStartSLOduration=2.072155293 podStartE2EDuration="2.072155293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:12.071684164 +0000 UTC m=+1352.021265066" watchObservedRunningTime="2026-09-09 16:36:12.072155293 +0000 UTC m=+1352.021736141"
Sep 09 16:36:23.052198 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:23.052162    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:36:23.052712 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:23.052480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:36:35.513394 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.513366    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:36:35.516364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.516347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.518710 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.518681    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-77176368\""
Sep 09 16:36:35.518848 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.518820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-77176368-dockercfg-zbfdc\""
Sep 09 16:36:35.524602 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.524576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:36:35.618075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.618046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkvd\" (UniqueName: \"kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.618231 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.618081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.719277 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.719249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkvd\" (UniqueName: \"kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.719401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.719285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.719978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.719943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.729325 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.729309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkvd\" (UniqueName: \"kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd\") pod \"payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:35.827409 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:35.827382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:36.099773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:36.099749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:36:36.101597 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:36:36.101571    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285a3ba9_53d9_4039_ad6f_fb94c20608e5.slice/crio-fcda41ef9be3654516d413ff6e517e0f53b1befad318b794c246f7bb6321483d WatchSource:0}: Error finding container fcda41ef9be3654516d413ff6e517e0f53b1befad318b794c246f7bb6321483d: Status 404 returned error can't find the container with id fcda41ef9be3654516d413ff6e517e0f53b1befad318b794c246f7bb6321483d
Sep 09 16:36:36.165141 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:36.165116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" event={"ID":"285a3ba9-53d9-4039-ad6f-fb94c20608e5","Type":"ContainerStarted","Data":"fcda41ef9be3654516d413ff6e517e0f53b1befad318b794c246f7bb6321483d"}
Sep 09 16:36:37.172497 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:37.172466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" event={"ID":"285a3ba9-53d9-4039-ad6f-fb94c20608e5","Type":"ContainerStarted","Data":"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"}
Sep 09 16:36:37.172898 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:37.172530    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:37.186770 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:37.186727    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" podStartSLOduration=2.186713674 podStartE2EDuration="2.186713674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:36:37.185642685 +0000 UTC m=+1377.135223532" watchObservedRunningTime="2026-09-09 16:36:37.186713674 +0000 UTC m=+1377.136294518"
Sep 09 16:36:48.179777 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:48.179747    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:36:53.699295 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:53.699245    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:36:55.030746 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.030715    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"]
Sep 09 16:36:55.034252 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.034233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:55.036267 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.036250    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jz5ff\""
Sep 09 16:36:55.036425 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.036409    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:36:55.040899 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.040848    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"]
Sep 09 16:36:55.079378 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.079348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5\" (UID: \"1d9928da-a013-4286-975c-4c883f7301d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:55.180629 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.180584    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5\" (UID: \"1d9928da-a013-4286-975c-4c883f7301d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:55.181269 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.181246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5\" (UID: \"1d9928da-a013-4286-975c-4c883f7301d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:55.345044 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.344953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:55.611942 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:55.611915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"]
Sep 09 16:36:55.613772 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:36:55.613748    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9928da_a013_4286_975c_4c883f7301d1.slice/crio-c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937 WatchSource:0}: Error finding container c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937: Status 404 returned error can't find the container with id c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937
Sep 09 16:36:56.268823 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:56.268796    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d9928da-a013-4286-975c-4c883f7301d1" containerID="3106c04a35ccdb244327a9cc1c99b677b7db7ef99fc4bf3c6b1779e81767f8bf" exitCode=0
Sep 09 16:36:56.269125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:56.268877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5" event={"ID":"1d9928da-a013-4286-975c-4c883f7301d1","Type":"ContainerDied","Data":"3106c04a35ccdb244327a9cc1c99b677b7db7ef99fc4bf3c6b1779e81767f8bf"}
Sep 09 16:36:56.269125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:56.268940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5" event={"ID":"1d9928da-a013-4286-975c-4c883f7301d1","Type":"ContainerStarted","Data":"c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937"}
Sep 09 16:36:57.400723 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:57.400703    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:36:57.498433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:57.498409    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca\") pod \"1d9928da-a013-4286-975c-4c883f7301d1\" (UID: \"1d9928da-a013-4286-975c-4c883f7301d1\") "
Sep 09 16:36:57.498806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:57.498781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d9928da-a013-4286-975c-4c883f7301d1" (UID: "1d9928da-a013-4286-975c-4c883f7301d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:36:57.599648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:57.599623    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9928da-a013-4286-975c-4c883f7301d1-maas-api-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:36:58.280890 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:58.280856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5" event={"ID":"1d9928da-a013-4286-975c-4c883f7301d1","Type":"ContainerDied","Data":"c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937"}
Sep 09 16:36:58.281066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:58.280901    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c094b35a1a187661c26ea9433d2f6dd88f599c9df9b567051c21204718d16937"
Sep 09 16:36:58.281066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:36:58.280863    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"
Sep 09 16:37:03.608392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.608359    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:37:03.608801 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.608561    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" podUID="5befa65f-3583-49ac-91e4-63bc51be35c7" containerName="payload-processing" containerID="cri-o://4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239" gracePeriod=30
Sep 09 16:37:03.614651 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.614626    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:37:03.614821 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.614803    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" podUID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" containerName="payload-pre-processing" containerID="cri-o://5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df" gracePeriod=30
Sep 09 16:37:03.876148 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.876126    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:37:03.903132 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.903100    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:37:03.954804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.954780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6w9p\" (UniqueName: \"kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p\") pod \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") "
Sep 09 16:37:03.954913 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.954807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume\") pod \"5befa65f-3583-49ac-91e4-63bc51be35c7\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") "
Sep 09 16:37:03.954913 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.954850    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume\") pod \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\" (UID: \"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df\") "
Sep 09 16:37:03.954913 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.954879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbj6b\" (UniqueName: \"kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b\") pod \"5befa65f-3583-49ac-91e4-63bc51be35c7\" (UID: \"5befa65f-3583-49ac-91e4-63bc51be35c7\") "
Sep 09 16:37:03.955257 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.955230    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" (UID: "37e2ed23-3f2e-40a5-917e-f3bc7a65f7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:03.955358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.955233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5befa65f-3583-49ac-91e4-63bc51be35c7" (UID: "5befa65f-3583-49ac-91e4-63bc51be35c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:03.956786 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.956767    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b" (OuterVolumeSpecName: "kube-api-access-bbj6b") pod "5befa65f-3583-49ac-91e4-63bc51be35c7" (UID: "5befa65f-3583-49ac-91e4-63bc51be35c7"). InnerVolumeSpecName "kube-api-access-bbj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:03.956842 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:03.956796    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p" (OuterVolumeSpecName: "kube-api-access-t6w9p") pod "37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" (UID: "37e2ed23-3f2e-40a5-917e-f3bc7a65f7df"). InnerVolumeSpecName "kube-api-access-t6w9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:04.056352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.056321    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6w9p\" (UniqueName: \"kubernetes.io/projected/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-kube-api-access-t6w9p\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.056352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.056347    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5befa65f-3583-49ac-91e4-63bc51be35c7-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.056352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.056358    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.056542 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.056366    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbj6b\" (UniqueName: \"kubernetes.io/projected/5befa65f-3583-49ac-91e4-63bc51be35c7-kube-api-access-bbj6b\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:04.310542 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.310510    2559 generic.go:356] "Generic (PLEG): container finished" podID="5befa65f-3583-49ac-91e4-63bc51be35c7" containerID="4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239" exitCode=0
Sep 09 16:37:04.310719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.310545    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" event={"ID":"5befa65f-3583-49ac-91e4-63bc51be35c7","Type":"ContainerDied","Data":"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"}
Sep 09 16:37:04.310719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.310574    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"
Sep 09 16:37:04.310719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.310624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn" event={"ID":"5befa65f-3583-49ac-91e4-63bc51be35c7","Type":"ContainerDied","Data":"aee521a88f3ad9ab814f7a7e6fb6d5de818773f631fdbc52a56f0df9973821a7"}
Sep 09 16:37:04.310719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.310642    2559 scope.go:117] "RemoveContainer" containerID="4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"
Sep 09 16:37:04.311936 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.311913    2559 generic.go:356] "Generic (PLEG): container finished" podID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" containerID="5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df" exitCode=0
Sep 09 16:37:04.312064 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.311985    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"
Sep 09 16:37:04.312064 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.311993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" event={"ID":"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df","Type":"ContainerDied","Data":"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"}
Sep 09 16:37:04.312064 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.312032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm" event={"ID":"37e2ed23-3f2e-40a5-917e-f3bc7a65f7df","Type":"ContainerDied","Data":"7e1f6a42dddd2d0a5e25b62046a98ba655b4a5c18eb6cce27de20117d48379e7"}
Sep 09 16:37:04.320867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.320849    2559 scope.go:117] "RemoveContainer" containerID="4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"
Sep 09 16:37:04.321218 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:04.321197    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239\": container with ID starting with 4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239 not found: ID does not exist" containerID="4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"
Sep 09 16:37:04.321415 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.321222    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239"} err="failed to get container status \"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239\": rpc error: code = NotFound desc = could not find container \"4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239\": container with ID starting with 4dd848d49b980eb379d601d532cb7e0c08ab6fae7130c614accf6097a5aa1239 not found: ID does not exist"
Sep 09 16:37:04.321415 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.321238    2559 scope.go:117] "RemoveContainer" containerID="5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"
Sep 09 16:37:04.328569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.328553    2559 scope.go:117] "RemoveContainer" containerID="5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"
Sep 09 16:37:04.328792 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:04.328776    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df\": container with ID starting with 5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df not found: ID does not exist" containerID="5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"
Sep 09 16:37:04.328849 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.328795    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df"} err="failed to get container status \"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df\": rpc error: code = NotFound desc = could not find container \"5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df\": container with ID starting with 5889c6275b771916c0d34aa62520c5ad1c721ea7d81feca684ac4c37caa7a5df not found: ID does not exist"
Sep 09 16:37:04.331984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.331948    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:37:04.333598 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.333581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-26563f-85b87f9fc7-trrmn"]
Sep 09 16:37:04.343074 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.343053    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:37:04.346785 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.346762    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-26563f-57594f87d6-hglpm"]
Sep 09 16:37:04.530926 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.530901    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" path="/var/lib/kubelet/pods/37e2ed23-3f2e-40a5-917e-f3bc7a65f7df/volumes"
Sep 09 16:37:04.531277 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:04.531263    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5befa65f-3583-49ac-91e4-63bc51be35c7" path="/var/lib/kubelet/pods/5befa65f-3583-49ac-91e4-63bc51be35c7/volumes"
Sep 09 16:37:05.040124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040093    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"]
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040448    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5befa65f-3583-49ac-91e4-63bc51be35c7" containerName="payload-processing"
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040459    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5befa65f-3583-49ac-91e4-63bc51be35c7" containerName="payload-processing"
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040469    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" containerName="payload-pre-processing"
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040474    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" containerName="payload-pre-processing"
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d9928da-a013-4286-975c-4c883f7301d1" containerName="revoke-keys"
Sep 09 16:37:05.040489 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040488    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d9928da-a013-4286-975c-4c883f7301d1" containerName="revoke-keys"
Sep 09 16:37:05.040668 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040536    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d9928da-a013-4286-975c-4c883f7301d1" containerName="revoke-keys"
Sep 09 16:37:05.040668 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040546    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e2ed23-3f2e-40a5-917e-f3bc7a65f7df" containerName="payload-pre-processing"
Sep 09 16:37:05.040668 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.040554    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5befa65f-3583-49ac-91e4-63bc51be35c7" containerName="payload-processing"
Sep 09 16:37:05.044931 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.044914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:05.047410 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.047344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:37:05.048154 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.048113    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jz5ff\""
Sep 09 16:37:05.050115 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.050086    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"]
Sep 09 16:37:05.166322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.166286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77176368-n576b\" (UID: \"60748354-e90a-4037-9ae8-700223b455d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:05.267463 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.267431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77176368-n576b\" (UID: \"60748354-e90a-4037-9ae8-700223b455d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:05.268052 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.268034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-77176368-n576b\" (UID: \"60748354-e90a-4037-9ae8-700223b455d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:05.356483 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.356408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:05.622480 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:05.622463    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"]
Sep 09 16:37:05.627054 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:05.626589    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60748354_e90a_4037_9ae8_700223b455d9.slice/crio-703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc WatchSource:0}: Error finding container 703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc: Status 404 returned error can't find the container with id 703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc
Sep 09 16:37:06.325826 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:06.325795    2559 generic.go:356] "Generic (PLEG): container finished" podID="60748354-e90a-4037-9ae8-700223b455d9" containerID="85e49aa82a6a58aeddac4584a89bf4d93698176a6df835cfcd15c1c846a3e083" exitCode=0
Sep 09 16:37:06.326200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:06.325883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b" event={"ID":"60748354-e90a-4037-9ae8-700223b455d9","Type":"ContainerDied","Data":"85e49aa82a6a58aeddac4584a89bf4d93698176a6df835cfcd15c1c846a3e083"}
Sep 09 16:37:06.326200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:06.325930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b" event={"ID":"60748354-e90a-4037-9ae8-700223b455d9","Type":"ContainerStarted","Data":"703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc"}
Sep 09 16:37:07.465896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.465869    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:07.589718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.589625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca\") pod \"60748354-e90a-4037-9ae8-700223b455d9\" (UID: \"60748354-e90a-4037-9ae8-700223b455d9\") "
Sep 09 16:37:07.590063 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.590040    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60748354-e90a-4037-9ae8-700223b455d9" (UID: "60748354-e90a-4037-9ae8-700223b455d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:07.658648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.658569    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"]
Sep 09 16:37:07.660162 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.660138    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-630acf-zdsz5"]
Sep 09 16:37:07.690593 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:07.690561    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60748354-e90a-4037-9ae8-700223b455d9-maas-api-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:08.337328 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.337299    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"
Sep 09 16:37:08.337497 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.337331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b" event={"ID":"60748354-e90a-4037-9ae8-700223b455d9","Type":"ContainerDied","Data":"703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc"}
Sep 09 16:37:08.337497 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.337398    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="703078e3087a9ab20b68f3146e8a6d29c6520fa21a996c1a2de656909d464cbc"
Sep 09 16:37:08.531588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.531556    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d9928da-a013-4286-975c-4c883f7301d1" path="/var/lib/kubelet/pods/1d9928da-a013-4286-975c-4c883f7301d1/volumes"
Sep 09 16:37:08.719478 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.719393    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" podUID="7b6b0115-5858-4d20-85c3-977c3b3e6821" containerName="main" containerID="cri-o://95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9" gracePeriod=45
Sep 09 16:37:08.966792 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:08.966767    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:37:09.105093 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105058    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9jls\" (UniqueName: \"kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105263 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105175    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105263 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105263 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105263 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105263 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105261    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache\") pod \"7b6b0115-5858-4d20-85c3-977c3b3e6821\" (UID: \"7b6b0115-5858-4d20-85c3-977c3b3e6821\") "
Sep 09 16:37:09.105521 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105487    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home" (OuterVolumeSpecName: "home") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:09.105521 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105497    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:09.105663 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.105640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:09.107401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.107375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm" (OuterVolumeSpecName: "dshm") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:37:09.107401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.107384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:37:09.107401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.107375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls" (OuterVolumeSpecName: "kube-api-access-t9jls") pod "7b6b0115-5858-4d20-85c3-977c3b3e6821" (UID: "7b6b0115-5858-4d20-85c3-977c3b3e6821"). InnerVolumeSpecName "kube-api-access-t9jls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:09.206472 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206437    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b6b0115-5858-4d20-85c3-977c3b3e6821-tls-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.206472 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206463    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-tmp-dir\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.206472 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206471    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-home\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.206472 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206478    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-dshm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.206723 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206486    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b6b0115-5858-4d20-85c3-977c3b3e6821-model-cache\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.206723 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.206495    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9jls\" (UniqueName: \"kubernetes.io/projected/7b6b0115-5858-4d20-85c3-977c3b3e6821-kube-api-access-t9jls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:09.344112 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.344074    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b6b0115-5858-4d20-85c3-977c3b3e6821" containerID="95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9" exitCode=0
Sep 09 16:37:09.344284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.344124    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"
Sep 09 16:37:09.344284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.344165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" event={"ID":"7b6b0115-5858-4d20-85c3-977c3b3e6821","Type":"ContainerDied","Data":"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"}
Sep 09 16:37:09.344284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.344212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d" event={"ID":"7b6b0115-5858-4d20-85c3-977c3b3e6821","Type":"ContainerDied","Data":"aa71bf8d17b103fa164c6923366cf2166fc3e4adb42b9d20c42473c74115bf89"}
Sep 09 16:37:09.344284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.344226    2559 scope.go:117] "RemoveContainer" containerID="95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"
Sep 09 16:37:09.352976 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.352940    2559 scope.go:117] "RemoveContainer" containerID="95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"
Sep 09 16:37:09.353256 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:09.353236    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9\": container with ID starting with 95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9 not found: ID does not exist" containerID="95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"
Sep 09 16:37:09.353303 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.353263    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9"} err="failed to get container status \"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9\": rpc error: code = NotFound desc = could not find container \"95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9\": container with ID starting with 95d0571f3c105ad7ee53e49e3b60dec663e7f7de2b85e698baf748d6bf5e5ce9 not found: ID does not exist"
Sep 09 16:37:09.363524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.363461    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:37:09.366903 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:09.366877    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-630acf/test-model-630acf-kserve-798cbd9475-6q22d"]
Sep 09 16:37:10.532240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:10.532208    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6b0115-5858-4d20-85c3-977c3b3e6821" path="/var/lib/kubelet/pods/7b6b0115-5858-4d20-85c3-977c3b3e6821/volumes"
Sep 09 16:37:14.932468 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932431    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"]
Sep 09 16:37:14.932816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932765    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6b0115-5858-4d20-85c3-977c3b3e6821" containerName="main"
Sep 09 16:37:14.932816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932775    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6b0115-5858-4d20-85c3-977c3b3e6821" containerName="main"
Sep 09 16:37:14.932816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932787    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60748354-e90a-4037-9ae8-700223b455d9" containerName="revoke-keys"
Sep 09 16:37:14.932816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932793    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="60748354-e90a-4037-9ae8-700223b455d9" containerName="revoke-keys"
Sep 09 16:37:14.932994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="60748354-e90a-4037-9ae8-700223b455d9" containerName="revoke-keys"
Sep 09 16:37:14.932994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.932863    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6b0115-5858-4d20-85c3-977c3b3e6821" containerName="main"
Sep 09 16:37:14.935709 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.935694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:14.938040 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.938020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jz5ff\""
Sep 09 16:37:14.938040 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.938034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:37:14.942353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:14.942331    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"]
Sep 09 16:37:15.061518 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:15.061491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl\" (UID: \"1d0a1930-6f17-49a9-870a-0dadc3a27112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:15.162734 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:15.162707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl\" (UID: \"1d0a1930-6f17-49a9-870a-0dadc3a27112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:15.163362 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:15.163342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl\" (UID: \"1d0a1930-6f17-49a9-870a-0dadc3a27112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:15.246842 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:15.246789    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:15.516893 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:15.516842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"]
Sep 09 16:37:15.518981 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:15.518943    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0a1930_6f17_49a9_870a_0dadc3a27112.slice/crio-0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d WatchSource:0}: Error finding container 0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d: Status 404 returned error can't find the container with id 0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d
Sep 09 16:37:16.381266 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.381227    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d0a1930-6f17-49a9-870a-0dadc3a27112" containerID="d50022437691b85b5c18a8f665aa12a8f3544ad17d12748ee6cfefa361652a83" exitCode=0
Sep 09 16:37:16.381688 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.381266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl" event={"ID":"1d0a1930-6f17-49a9-870a-0dadc3a27112","Type":"ContainerDied","Data":"d50022437691b85b5c18a8f665aa12a8f3544ad17d12748ee6cfefa361652a83"}
Sep 09 16:37:16.381688 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.381336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl" event={"ID":"1d0a1930-6f17-49a9-870a-0dadc3a27112","Type":"ContainerStarted","Data":"0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d"}
Sep 09 16:37:16.424150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.424123    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:37:16.424336 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.424314    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" podUID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" containerName="payload-pre-processing" containerID="cri-o://0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816" gracePeriod=30
Sep 09 16:37:16.625464 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.625435    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"]
Sep 09 16:37:16.627882 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.627858    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-77176368-n576b"]
Sep 09 16:37:16.664207 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.664189    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:37:16.776013 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.775981    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnkvd\" (UniqueName: \"kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd\") pod \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") "
Sep 09 16:37:16.776126 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.776109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume\") pod \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\" (UID: \"285a3ba9-53d9-4039-ad6f-fb94c20608e5\") "
Sep 09 16:37:16.776446 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.776423    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "285a3ba9-53d9-4039-ad6f-fb94c20608e5" (UID: "285a3ba9-53d9-4039-ad6f-fb94c20608e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:16.777806 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.777787    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd" (OuterVolumeSpecName: "kube-api-access-hnkvd") pod "285a3ba9-53d9-4039-ad6f-fb94c20608e5" (UID: "285a3ba9-53d9-4039-ad6f-fb94c20608e5"). InnerVolumeSpecName "kube-api-access-hnkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:16.876715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.876664    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkvd\" (UniqueName: \"kubernetes.io/projected/285a3ba9-53d9-4039-ad6f-fb94c20608e5-kube-api-access-hnkvd\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:16.876715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:16.876684    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/285a3ba9-53d9-4039-ad6f-fb94c20608e5-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:17.388307 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.388274    2559 generic.go:356] "Generic (PLEG): container finished" podID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" containerID="0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816" exitCode=0
Sep 09 16:37:17.388685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.388329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"
Sep 09 16:37:17.388685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.388366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" event={"ID":"285a3ba9-53d9-4039-ad6f-fb94c20608e5","Type":"ContainerDied","Data":"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"}
Sep 09 16:37:17.388685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.388429    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk" event={"ID":"285a3ba9-53d9-4039-ad6f-fb94c20608e5","Type":"ContainerDied","Data":"fcda41ef9be3654516d413ff6e517e0f53b1befad318b794c246f7bb6321483d"}
Sep 09 16:37:17.388685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.388457    2559 scope.go:117] "RemoveContainer" containerID="0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"
Sep 09 16:37:17.398865 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.398847    2559 scope.go:117] "RemoveContainer" containerID="0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"
Sep 09 16:37:17.399284 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:17.399261    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816\": container with ID starting with 0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816 not found: ID does not exist" containerID="0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"
Sep 09 16:37:17.399348 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.399294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816"} err="failed to get container status \"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816\": rpc error: code = NotFound desc = could not find container \"0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816\": container with ID starting with 0482be4a21f8e00687599a0b87cdaae0d20aef22483069b8d74466a8fc2ec816 not found: ID does not exist"
Sep 09 16:37:17.412167 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.412140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:37:17.417246 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.417221    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-77176368-ddbf95998-whvqk"]
Sep 09 16:37:17.525070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.525045    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:17.682661 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.682601    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca\") pod \"1d0a1930-6f17-49a9-870a-0dadc3a27112\" (UID: \"1d0a1930-6f17-49a9-870a-0dadc3a27112\") "
Sep 09 16:37:17.683021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.682990    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d0a1930-6f17-49a9-870a-0dadc3a27112" (UID: "1d0a1930-6f17-49a9-870a-0dadc3a27112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:17.783684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:17.783661    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0a1930-6f17-49a9-870a-0dadc3a27112-maas-api-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:18.396053 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:18.396010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl" event={"ID":"1d0a1930-6f17-49a9-870a-0dadc3a27112","Type":"ContainerDied","Data":"0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d"}
Sep 09 16:37:18.396053 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:18.396041    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"
Sep 09 16:37:18.396485 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:18.396111    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca48d188c02c2f2d4a3688aa0f7cccbe589fc6cae29b46c69ccd1bc68ef1e1d"
Sep 09 16:37:18.532239 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:18.532207    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" path="/var/lib/kubelet/pods/285a3ba9-53d9-4039-ad6f-fb94c20608e5/volumes"
Sep 09 16:37:18.532524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:18.532510    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60748354-e90a-4037-9ae8-700223b455d9" path="/var/lib/kubelet/pods/60748354-e90a-4037-9ae8-700223b455d9/volumes"
Sep 09 16:37:25.336166 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.336134    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:37:25.336622 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.336370    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" podUID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" containerName="payload-processing" containerID="cri-o://de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5" gracePeriod=30
Sep 09 16:37:25.344364 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.344338    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:37:25.344730 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.344679    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" podUID="e61dfbec-f5be-4e32-945f-5515d11d373e" containerName="payload-pre-processing" containerID="cri-o://646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3" gracePeriod=30
Sep 09 16:37:25.473784 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.473754    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"]
Sep 09 16:37:25.476179 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.476152    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34141f-kcmvl"]
Sep 09 16:37:25.593696 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.593637    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:37:25.614062 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.614041    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:37:25.743201 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743153    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume\") pod \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") "
Sep 09 16:37:25.743382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743222    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn4dz\" (UniqueName: \"kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz\") pod \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\" (UID: \"b3277f19-f369-4a46-9cd2-31e67e5fe9da\") "
Sep 09 16:37:25.743382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743251    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsqlk\" (UniqueName: \"kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk\") pod \"e61dfbec-f5be-4e32-945f-5515d11d373e\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") "
Sep 09 16:37:25.743382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743325    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume\") pod \"e61dfbec-f5be-4e32-945f-5515d11d373e\" (UID: \"e61dfbec-f5be-4e32-945f-5515d11d373e\") "
Sep 09 16:37:25.743611 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3277f19-f369-4a46-9cd2-31e67e5fe9da" (UID: "b3277f19-f369-4a46-9cd2-31e67e5fe9da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:25.743767 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.743745    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e61dfbec-f5be-4e32-945f-5515d11d373e" (UID: "e61dfbec-f5be-4e32-945f-5515d11d373e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:37:25.745181 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.745158    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz" (OuterVolumeSpecName: "kube-api-access-zn4dz") pod "b3277f19-f369-4a46-9cd2-31e67e5fe9da" (UID: "b3277f19-f369-4a46-9cd2-31e67e5fe9da"). InnerVolumeSpecName "kube-api-access-zn4dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:25.745302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.745285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk" (OuterVolumeSpecName: "kube-api-access-vsqlk") pod "e61dfbec-f5be-4e32-945f-5515d11d373e" (UID: "e61dfbec-f5be-4e32-945f-5515d11d373e"). InnerVolumeSpecName "kube-api-access-vsqlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:37:25.844439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.844356    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dfbec-f5be-4e32-945f-5515d11d373e-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.844439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.844387    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3277f19-f369-4a46-9cd2-31e67e5fe9da-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.844439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.844399    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4dz\" (UniqueName: \"kubernetes.io/projected/b3277f19-f369-4a46-9cd2-31e67e5fe9da-kube-api-access-zn4dz\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:25.844439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:25.844409    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqlk\" (UniqueName: \"kubernetes.io/projected/e61dfbec-f5be-4e32-945f-5515d11d373e-kube-api-access-vsqlk\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:37:26.439550 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.439522    2559 generic.go:356] "Generic (PLEG): container finished" podID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" containerID="de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5" exitCode=0
Sep 09 16:37:26.439984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.439566    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"
Sep 09 16:37:26.439984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.439612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" event={"ID":"b3277f19-f369-4a46-9cd2-31e67e5fe9da","Type":"ContainerDied","Data":"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"}
Sep 09 16:37:26.439984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.439670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm" event={"ID":"b3277f19-f369-4a46-9cd2-31e67e5fe9da","Type":"ContainerDied","Data":"dd82d345820f67d9d779e57c78f71e475eea3fc5bca77335ccd8b45088215037"}
Sep 09 16:37:26.439984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.439703    2559 scope.go:117] "RemoveContainer" containerID="de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"
Sep 09 16:37:26.441059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.440857    2559 generic.go:356] "Generic (PLEG): container finished" podID="e61dfbec-f5be-4e32-945f-5515d11d373e" containerID="646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3" exitCode=0
Sep 09 16:37:26.441059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.440889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"
Sep 09 16:37:26.441059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.440928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" event={"ID":"e61dfbec-f5be-4e32-945f-5515d11d373e","Type":"ContainerDied","Data":"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"}
Sep 09 16:37:26.441059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.440988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm" event={"ID":"e61dfbec-f5be-4e32-945f-5515d11d373e","Type":"ContainerDied","Data":"f6bfea6f25ce4c4c85cd9c6b8baad845e0078d672484c2131c133b1501f48d01"}
Sep 09 16:37:26.448839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.448792    2559 scope.go:117] "RemoveContainer" containerID="de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"
Sep 09 16:37:26.449110 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:26.449093    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5\": container with ID starting with de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5 not found: ID does not exist" containerID="de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"
Sep 09 16:37:26.449173 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.449117    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5"} err="failed to get container status \"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5\": rpc error: code = NotFound desc = could not find container \"de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5\": container with ID starting with de1a7d43b5ead389370e4b51157cebcd0393feb658ca8edbfab69a4a893c9de5 not found: ID does not exist"
Sep 09 16:37:26.449173 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.449133    2559 scope.go:117] "RemoveContainer" containerID="646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"
Sep 09 16:37:26.456666 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.456647    2559 scope.go:117] "RemoveContainer" containerID="646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"
Sep 09 16:37:26.456913 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:37:26.456897    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3\": container with ID starting with 646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3 not found: ID does not exist" containerID="646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"
Sep 09 16:37:26.456978 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.456920    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3"} err="failed to get container status \"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3\": rpc error: code = NotFound desc = could not find container \"646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3\": container with ID starting with 646053f444e2f2ab5743c74b3f1f9673e298a3e22f1c54c9ed8864af7fee0ee3 not found: ID does not exist"
Sep 09 16:37:26.464409 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.464389    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:37:26.468090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.468070    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34141f-6d444ff8df-gz5sm"]
Sep 09 16:37:26.477504 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.477484    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:37:26.479591 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.479573    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34141f-ff7fdf65d-4thwm"]
Sep 09 16:37:26.530466 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.530447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0a1930-6f17-49a9-870a-0dadc3a27112" path="/var/lib/kubelet/pods/1d0a1930-6f17-49a9-870a-0dadc3a27112/volumes"
Sep 09 16:37:26.530777 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.530765    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" path="/var/lib/kubelet/pods/b3277f19-f369-4a46-9cd2-31e67e5fe9da/volumes"
Sep 09 16:37:26.531090 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:26.531079    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e61dfbec-f5be-4e32-945f-5515d11d373e" path="/var/lib/kubelet/pods/e61dfbec-f5be-4e32-945f-5515d11d373e/volumes"
Sep 09 16:37:36.944169 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944082    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:37:36.944623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944571    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e61dfbec-f5be-4e32-945f-5515d11d373e" containerName="payload-pre-processing"
Sep 09 16:37:36.944623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944589    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e61dfbec-f5be-4e32-945f-5515d11d373e" containerName="payload-pre-processing"
Sep 09 16:37:36.944623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944613    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" containerName="payload-processing"
Sep 09 16:37:36.944623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" containerName="payload-processing"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944639    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0a1930-6f17-49a9-870a-0dadc3a27112" containerName="revoke-keys"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944647    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0a1930-6f17-49a9-870a-0dadc3a27112" containerName="revoke-keys"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944665    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" containerName="payload-pre-processing"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944673    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" containerName="payload-pre-processing"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3277f19-f369-4a46-9cd2-31e67e5fe9da" containerName="payload-processing"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944766    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0a1930-6f17-49a9-870a-0dadc3a27112" containerName="revoke-keys"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944777    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="285a3ba9-53d9-4039-ad6f-fb94c20608e5" containerName="payload-pre-processing"
Sep 09 16:37:36.944831 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.944787    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e61dfbec-f5be-4e32-945f-5515d11d373e" containerName="payload-pre-processing"
Sep 09 16:37:36.949984 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.949558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:36.952268 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.952240    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-331bf8-serving-cert\""
Sep 09 16:37:36.953165 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.953138    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 16:37:36.953299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.953211    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k96v8\""
Sep 09 16:37:36.953299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.953207    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:37:36.953397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.953255    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 16:37:36.960532 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:36.960507    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:37:37.049763 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.049723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.049949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.049785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.049949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.049808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.049949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.049825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mks4m\" (UniqueName: \"kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.150616 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.150570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.150814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.150637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.150885 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.150814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.150885 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.150863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mks4m\" (UniqueName: \"kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.151345 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.151322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.153119 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.153093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.153245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.153223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.158536 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.158505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mks4m\" (UniqueName: \"kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m\") pod \"maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.261031 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.260925    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:37.502633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.502597    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:37:37.507199 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.507176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.509841 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.509813    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-331bf8-dockercfg-pqbbt\""
Sep 09 16:37:37.510016 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.509862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-331bf8\""
Sep 09 16:37:37.511260 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.511190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:37:37.533683 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.533644    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:37:37.537512 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:37.537479    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba89743d_3a8d_45fb_a01b_783e9a798cdc.slice/crio-088999a7d0cc42f39bfc2ea259dac47565ba727f065ecd5753ab6c4127347b0e WatchSource:0}: Error finding container 088999a7d0cc42f39bfc2ea259dac47565ba727f065ecd5753ab6c4127347b0e: Status 404 returned error can't find the container with id 088999a7d0cc42f39bfc2ea259dac47565ba727f065ecd5753ab6c4127347b0e
Sep 09 16:37:37.539267 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.539243    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:37:37.555988 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.555923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492qj\" (UniqueName: \"kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.556206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.556099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.657129 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.657096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492qj\" (UniqueName: \"kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.657302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.657160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.658053 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.658031    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.665339 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.665310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492qj\" (UniqueName: \"kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj\") pod \"payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:37.820065 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:37.820033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:38.090839 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.090754    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:37:38.093903 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:38.093861    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3520d030_07ef_470a_b624_7f52dd765be3.slice/crio-db09f257833e2a62d5601351bf0fe2087f3daa8565b0f2af50b79052503eee0f WatchSource:0}: Error finding container db09f257833e2a62d5601351bf0fe2087f3daa8565b0f2af50b79052503eee0f: Status 404 returned error can't find the container with id db09f257833e2a62d5601351bf0fe2087f3daa8565b0f2af50b79052503eee0f
Sep 09 16:37:38.502049 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.501951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" event={"ID":"3520d030-07ef-470a-b624-7f52dd765be3","Type":"ContainerStarted","Data":"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"}
Sep 09 16:37:38.502049 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.502029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" event={"ID":"3520d030-07ef-470a-b624-7f52dd765be3","Type":"ContainerStarted","Data":"db09f257833e2a62d5601351bf0fe2087f3daa8565b0f2af50b79052503eee0f"}
Sep 09 16:37:38.502257 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.502052    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:38.503306 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.503275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" event={"ID":"ba89743d-3a8d-45fb-a01b-783e9a798cdc","Type":"ContainerStarted","Data":"088999a7d0cc42f39bfc2ea259dac47565ba727f065ecd5753ab6c4127347b0e"}
Sep 09 16:37:38.517387 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:38.517331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" podStartSLOduration=1.517313168 podStartE2EDuration="1.517313168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:38.515927872 +0000 UTC m=+1438.465508730" watchObservedRunningTime="2026-09-09 16:37:38.517313168 +0000 UTC m=+1438.466894005"
Sep 09 16:37:39.509792 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:39.509750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" event={"ID":"ba89743d-3a8d-45fb-a01b-783e9a798cdc","Type":"ContainerStarted","Data":"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"}
Sep 09 16:37:39.510275 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:39.509950    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:39.527304 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:39.527238    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" podStartSLOduration=2.543958263 podStartE2EDuration="3.527221043s" totalImagesPullingTime="983.26278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:39.525566799 +0000 UTC m=+1439.475147701" watchObservedRunningTime="2026-09-09 16:37:39.527221043 +0000 UTC m=+1439.476801888"
Sep 09 16:37:41.785890 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.785852    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:37:41.789238 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.789214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:41.793644 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.793621    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2bdf45\""
Sep 09 16:37:41.798412 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.798386    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:37:41.851354 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.851317    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:37:41.854576 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.854559    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.857258 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.857237    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2bdf45-dockercfg-df8m5\""
Sep 09 16:37:41.862872 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.862846    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:37:41.894735 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.894690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm84b\" (UniqueName: \"kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:41.894735 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.894730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.894949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.894746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5d4f\" (UniqueName: \"kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.894949 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.894824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:41.995977 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.995921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.995977 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.995979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5d4f\" (UniqueName: \"kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.996222 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.996021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:41.996222 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.996144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm84b\" (UniqueName: \"kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:41.996731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.996711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:41.996764 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:41.996736    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:42.003929 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.003896    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm84b\" (UniqueName: \"kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b\") pod \"payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:42.004116 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.003892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5d4f\" (UniqueName: \"kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f\") pod \"payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:42.100542 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.100448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:42.166528 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.166489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:42.373399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.373375    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:37:42.375556 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:42.375530    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb711e8_44e5_4fa0_99f6_d520ba321a84.slice/crio-a4c90fde96b84d82ce4f17994638b9caf1751e3633fadfe794838e3502cfde15 WatchSource:0}: Error finding container a4c90fde96b84d82ce4f17994638b9caf1751e3633fadfe794838e3502cfde15: Status 404 returned error can't find the container with id a4c90fde96b84d82ce4f17994638b9caf1751e3633fadfe794838e3502cfde15
Sep 09 16:37:42.443267 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.443226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:37:42.446079 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:37:42.446055    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d89d361_a3da_40bc_9f40_81a9ad657862.slice/crio-85b61ca5a5e67044b93c51b0ddb5672e18aa58a20e772685fde4f1b933342132 WatchSource:0}: Error finding container 85b61ca5a5e67044b93c51b0ddb5672e18aa58a20e772685fde4f1b933342132: Status 404 returned error can't find the container with id 85b61ca5a5e67044b93c51b0ddb5672e18aa58a20e772685fde4f1b933342132
Sep 09 16:37:42.531810 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:37:42.532014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531831    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:42.532014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531858    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" event={"ID":"6d89d361-a3da-40bc-9f40-81a9ad657862","Type":"ContainerStarted","Data":"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"}
Sep 09 16:37:42.532014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" event={"ID":"6d89d361-a3da-40bc-9f40-81a9ad657862","Type":"ContainerStarted","Data":"85b61ca5a5e67044b93c51b0ddb5672e18aa58a20e772685fde4f1b933342132"}
Sep 09 16:37:42.532014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" event={"ID":"8cb711e8-44e5-4fa0-99f6-d520ba321a84","Type":"ContainerStarted","Data":"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"}
Sep 09 16:37:42.532014 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.531918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" event={"ID":"8cb711e8-44e5-4fa0-99f6-d520ba321a84","Type":"ContainerStarted","Data":"a4c90fde96b84d82ce4f17994638b9caf1751e3633fadfe794838e3502cfde15"}
Sep 09 16:37:42.541554 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.541502    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" podStartSLOduration=1.541484434 podStartE2EDuration="1.541484434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:42.540923144 +0000 UTC m=+1442.490503990" watchObservedRunningTime="2026-09-09 16:37:42.541484434 +0000 UTC m=+1442.491065279"
Sep 09 16:37:42.555356 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:42.555187    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" podStartSLOduration=1.555164912 podStartE2EDuration="1.555164912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:37:42.554458268 +0000 UTC m=+1442.504039141" watchObservedRunningTime="2026-09-09 16:37:42.555164912 +0000 UTC m=+1442.504745759"
Sep 09 16:37:45.520354 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:45.520324    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:37:49.512028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:49.511997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:37:53.536417 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:53.536385    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:37:54.541634 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:37:54.541602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:38:05.979003 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:05.978953    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:38:05.982698 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:05.982676    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:05.985690 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:05.985669    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-50fcbc-serving-cert\""
Sep 09 16:38:05.992299 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:05.992278    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:38:06.118623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.118585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5tl\" (UniqueName: \"kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.118623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.118628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.118859 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.118665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.118859 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.118722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.219588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.219544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5tl\" (UniqueName: \"kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.219769 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.219592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.219769 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.219629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.219769 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.219651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.219921 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:38:06.219765    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-serving-cert: secret "maas-api-e2e-shared-b-w3-50fcbc-serving-cert" not found
Sep 09 16:38:06.219921 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:38:06.219845    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls podName:75310ee7-5020-4aed-98d1-e2b0f9690826 nodeName:}" failed. No retries permitted until 2026-09-09 16:38:06.71982364 +0000 UTC m=+1466.669404474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls") pod "maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" (UID: "75310ee7-5020-4aed-98d1-e2b0f9690826") : secret "maas-api-e2e-shared-b-w3-50fcbc-serving-cert" not found
Sep 09 16:38:06.220401 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.220378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.221932 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.221912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.227517 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.227496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5tl\" (UniqueName: \"kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.723595 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.723547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.725898 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.725868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:06.894402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:06.894359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:07.162679 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:07.162649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:38:07.165473 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:38:07.165449    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75310ee7_5020_4aed_98d1_e2b0f9690826.slice/crio-b3353de3e623ec7721288d8624dcc91887cd9e5685f98796935765ada91f2356 WatchSource:0}: Error finding container b3353de3e623ec7721288d8624dcc91887cd9e5685f98796935765ada91f2356: Status 404 returned error can't find the container with id b3353de3e623ec7721288d8624dcc91887cd9e5685f98796935765ada91f2356
Sep 09 16:38:07.653346 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:07.653306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" event={"ID":"75310ee7-5020-4aed-98d1-e2b0f9690826","Type":"ContainerStarted","Data":"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"}
Sep 09 16:38:07.653513 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:07.653364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" event={"ID":"75310ee7-5020-4aed-98d1-e2b0f9690826","Type":"ContainerStarted","Data":"b3353de3e623ec7721288d8624dcc91887cd9e5685f98796935765ada91f2356"}
Sep 09 16:38:07.653513 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:07.653415    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:07.673392 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:07.673325    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" podStartSLOduration=2.337715353 podStartE2EDuration="2.673310794s" totalImagesPullingTime="335.595441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:07.672500528 +0000 UTC m=+1467.622081385" watchObservedRunningTime="2026-09-09 16:38:07.673310794 +0000 UTC m=+1467.622891640"
Sep 09 16:38:11.066906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.066872    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:38:11.070720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.070698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.073034 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.073013    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-1d59dd-serving-cert\""
Sep 09 16:38:11.080703 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.080681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:38:11.156569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.156531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.156569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.156569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqd47\" (UniqueName: \"kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.156773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.156617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.156773 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.156686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.257277 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.257248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.257398 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.257286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.257398 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.257355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.257398 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.257373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqd47\" (UniqueName: \"kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.257498 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:38:11.257462    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-serving-cert: secret "maas-api-e2e-shared-b-w2-1d59dd-serving-cert" not found
Sep 09 16:38:11.257532 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:38:11.257520    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls podName:9b2c3d7c-7a7c-4dde-a931-13dccd71b090 nodeName:}" failed. No retries permitted until 2026-09-09 16:38:11.757505513 +0000 UTC m=+1471.707086338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls") pod "maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" (UID: "9b2c3d7c-7a7c-4dde-a931-13dccd71b090") : secret "maas-api-e2e-shared-b-w2-1d59dd-serving-cert" not found
Sep 09 16:38:11.257870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.257852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.259790 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.259767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.265001 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.264978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqd47\" (UniqueName: \"kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.760660 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.760605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.762424 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.762398    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:38:11.763102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.763077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:11.765920 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.765904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.768294 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.768273    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1d59dd-dockercfg-874xt\""
Sep 09 16:38:11.768399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.768280    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1d59dd\""
Sep 09 16:38:11.777424 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.777406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:38:11.861721 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.861695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wgw\" (UniqueName: \"kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.861850 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.861755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.962186 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.962147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wgw\" (UniqueName: \"kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.962363 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.962209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.962858 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.962838    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.969420 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.969398    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wgw\" (UniqueName: \"kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw\") pod \"payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:11.983334 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:11.983302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:12.081385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.081351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:12.254594 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.254570    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:38:12.255949 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:38:12.255926    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2c3d7c_7a7c_4dde_a931_13dccd71b090.slice/crio-8e6801affb25f06b6205d45c1dedafd04e64982f78afbba76866592233274bef WatchSource:0}: Error finding container 8e6801affb25f06b6205d45c1dedafd04e64982f78afbba76866592233274bef: Status 404 returned error can't find the container with id 8e6801affb25f06b6205d45c1dedafd04e64982f78afbba76866592233274bef
Sep 09 16:38:12.348830 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.348682    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:38:12.351488 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:38:12.351463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc673fe08_f9e5_419f_8abf_42968ac7c39b.slice/crio-35418d9919e11b258f58ed84c99831e727622cccf657f068e1b164790aded453 WatchSource:0}: Error finding container 35418d9919e11b258f58ed84c99831e727622cccf657f068e1b164790aded453: Status 404 returned error can't find the container with id 35418d9919e11b258f58ed84c99831e727622cccf657f068e1b164790aded453
Sep 09 16:38:12.680315 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.680242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" event={"ID":"9b2c3d7c-7a7c-4dde-a931-13dccd71b090","Type":"ContainerStarted","Data":"8e6801affb25f06b6205d45c1dedafd04e64982f78afbba76866592233274bef"}
Sep 09 16:38:12.681843 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.681821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" event={"ID":"c673fe08-f9e5-419f-8abf-42968ac7c39b","Type":"ContainerStarted","Data":"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"}
Sep 09 16:38:12.681979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.681859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" event={"ID":"c673fe08-f9e5-419f-8abf-42968ac7c39b","Type":"ContainerStarted","Data":"35418d9919e11b258f58ed84c99831e727622cccf657f068e1b164790aded453"}
Sep 09 16:38:12.682028 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.681979    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:12.696866 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:12.696824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" podStartSLOduration=1.696810053 podStartE2EDuration="1.696810053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:12.694804377 +0000 UTC m=+1472.644385224" watchObservedRunningTime="2026-09-09 16:38:12.696810053 +0000 UTC m=+1472.646390899"
Sep 09 16:38:13.664139 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:13.664117    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:38:13.690129 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:13.690100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" event={"ID":"9b2c3d7c-7a7c-4dde-a931-13dccd71b090","Type":"ContainerStarted","Data":"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"}
Sep 09 16:38:13.690270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:13.690167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:13.705730 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:13.705688    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" podStartSLOduration=1.496815941 podStartE2EDuration="2.705675983s" totalImagesPullingTime="1.208860042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:13.704614848 +0000 UTC m=+1473.654195695" watchObservedRunningTime="2026-09-09 16:38:13.705675983 +0000 UTC m=+1473.655256852"
Sep 09 16:38:19.700809 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:19.700780    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:38:23.692430 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:23.692398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:38:41.451843 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:41.451811    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:38:41.463344 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:41.463316    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:38:58.286798 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.286757    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:38:58.290721 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.290700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.294235 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.294212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"openshift-service-ca.crt\""
Sep 09 16:38:58.294381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.294317    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"auth-test-model-50fcbc-kserve-self-signed-certs\""
Sep 09 16:38:58.294381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.294356    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"kube-root-ca.crt\""
Sep 09 16:38:58.294381 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.294366    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-50fcbc\"/\"default-dockercfg-qltth\""
Sep 09 16:38:58.299923 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.299896    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:38:58.380609 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.380609 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbljl\" (UniqueName: \"kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.380812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.380812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.380923 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.380923 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.380900    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482270 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbljl\" (UniqueName: \"kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482453 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482748 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482792 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.482860 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.482789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.484650 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.484630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.485017 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.485001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.489546 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.489516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbljl\" (UniqueName: \"kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl\") pod \"auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") " pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.601036 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.600908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:38:58.873479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.873453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:38:58.875903 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:38:58.875872    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4621d44f_836f_471b_97b6_6550ac34bafe.slice/crio-4eef274ac0467f7b17ccf2f0a63e9cf6bec60a75a088ceec13f16a8563286ae7 WatchSource:0}: Error finding container 4eef274ac0467f7b17ccf2f0a63e9cf6bec60a75a088ceec13f16a8563286ae7: Status 404 returned error can't find the container with id 4eef274ac0467f7b17ccf2f0a63e9cf6bec60a75a088ceec13f16a8563286ae7
Sep 09 16:38:58.921939 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:58.921906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" event={"ID":"4621d44f-836f-471b-97b6-6550ac34bafe","Type":"ContainerStarted","Data":"4eef274ac0467f7b17ccf2f0a63e9cf6bec60a75a088ceec13f16a8563286ae7"}
Sep 09 16:38:59.929283 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:59.929245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" event={"ID":"4621d44f-836f-471b-97b6-6550ac34bafe","Type":"ContainerStarted","Data":"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"}
Sep 09 16:38:59.943804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:38:59.943739    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" podStartSLOduration=1.943716888 podStartE2EDuration="1.943716888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:38:59.94329864 +0000 UTC m=+1519.892879489" watchObservedRunningTime="2026-09-09 16:38:59.943716888 +0000 UTC m=+1519.893297735"
Sep 09 16:39:08.601149 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:39:08.601052    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:39:08.601149 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:39:08.601116    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:39:08.613555 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:39:08.613520    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:39:08.991523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:39:08.991432    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:39:40.686469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:39:40.686435    2559 scope.go:117] "RemoveContainer" containerID="9aeaf71993da48ff56ddb06ffa1731e1144f5f1236f194423d4faba5a668fe3f"
Sep 09 16:41:05.776196 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:05.776162    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:41:20.795065 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:20.795025    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" podUID="4621d44f-836f-471b-97b6-6550ac34bafe" containerName="main" containerID="cri-o://40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41" gracePeriod=45
Sep 09 16:41:21.033860 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.033836    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:41:21.044305 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044286    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbljl\" (UniqueName: \"kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044390 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044318    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044390 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044390 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044359    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044567 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm\") pod \"4621d44f-836f-471b-97b6-6550ac34bafe\" (UID: \"4621d44f-836f-471b-97b6-6550ac34bafe\") "
Sep 09 16:41:21.044618 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home" (OuterVolumeSpecName: "home") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:21.044618 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache" (OuterVolumeSpecName: "model-cache") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:21.044727 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044694    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:21.044857 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044840    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-model-cache\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.044911 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044863    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-home\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.044911 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.044877    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-tmp-dir\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.046446 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.046398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:41:21.046446 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.046425    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm" (OuterVolumeSpecName: "dshm") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:41:21.046588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.046462    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl" (OuterVolumeSpecName: "kube-api-access-bbljl") pod "4621d44f-836f-471b-97b6-6550ac34bafe" (UID: "4621d44f-836f-471b-97b6-6550ac34bafe"). InnerVolumeSpecName "kube-api-access-bbljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:41:21.146194 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.146173    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbljl\" (UniqueName: \"kubernetes.io/projected/4621d44f-836f-471b-97b6-6550ac34bafe-kube-api-access-bbljl\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.146194 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.146194    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4621d44f-836f-471b-97b6-6550ac34bafe-tls-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.146350 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.146203    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4621d44f-836f-471b-97b6-6550ac34bafe-dshm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:41:21.652807 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.652775    2559 generic.go:356] "Generic (PLEG): container finished" podID="4621d44f-836f-471b-97b6-6550ac34bafe" containerID="40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41" exitCode=0
Sep 09 16:41:21.653011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.652836    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"
Sep 09 16:41:21.653011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.652859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" event={"ID":"4621d44f-836f-471b-97b6-6550ac34bafe","Type":"ContainerDied","Data":"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"}
Sep 09 16:41:21.653011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.652906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf" event={"ID":"4621d44f-836f-471b-97b6-6550ac34bafe","Type":"ContainerDied","Data":"4eef274ac0467f7b17ccf2f0a63e9cf6bec60a75a088ceec13f16a8563286ae7"}
Sep 09 16:41:21.653011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.652923    2559 scope.go:117] "RemoveContainer" containerID="40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"
Sep 09 16:41:21.662059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.662038    2559 scope.go:117] "RemoveContainer" containerID="40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"
Sep 09 16:41:21.662297 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:41:21.662279    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41\": container with ID starting with 40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41 not found: ID does not exist" containerID="40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"
Sep 09 16:41:21.662355 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.662308    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41"} err="failed to get container status \"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41\": rpc error: code = NotFound desc = could not find container \"40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41\": container with ID starting with 40a9c4d5b241dd0418c1618a81c7dd835108f514e008ab7ee76e2ab85f046d41 not found: ID does not exist"
Sep 09 16:41:21.675118 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.675088    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:41:21.678840 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:21.678820    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-50fcbc/auth-test-model-50fcbc-kserve-58d9dcfc75-29lwf"]
Sep 09 16:41:22.531561 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:41:22.531528    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4621d44f-836f-471b-97b6-6550ac34bafe" path="/var/lib/kubelet/pods/4621d44f-836f-471b-97b6-6550ac34bafe/volumes"
Sep 09 16:42:19.794193 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.794118    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:42:19.794623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.794494    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4621d44f-836f-471b-97b6-6550ac34bafe" containerName="main"
Sep 09 16:42:19.794623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.794504    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4621d44f-836f-471b-97b6-6550ac34bafe" containerName="main"
Sep 09 16:42:19.794623 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.794568    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4621d44f-836f-471b-97b6-6550ac34bafe" containerName="main"
Sep 09 16:42:19.797608 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.797589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:19.799914 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.799896    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-14e637\""
Sep 09 16:42:19.800103 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.800088    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-14e637-dockercfg-rfdz6\""
Sep 09 16:42:19.804249 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.804229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:42:19.897645 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.897622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:19.897645 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.897648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snth\" (UniqueName: \"kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:19.998394 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.998365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:19.998394 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.998397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snth\" (UniqueName: \"kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:19.999079 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:19.999061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:20.007055 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.007026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snth\" (UniqueName: \"kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth\") pod \"payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:20.108999 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.108899    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:20.377095 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.375686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:42:20.936493 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.936458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" event={"ID":"ce2791db-4a0f-4848-bdf0-ccdefc1010cd","Type":"ContainerStarted","Data":"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"}
Sep 09 16:42:20.936895 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.936516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" event={"ID":"ce2791db-4a0f-4848-bdf0-ccdefc1010cd","Type":"ContainerStarted","Data":"90fcb239580ed7d78bcd943ddbe51a7eb30ce6ee01c08051779b5f60b815a067"}
Sep 09 16:42:20.936895 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.936564    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:20.950786 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:20.950726    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" podStartSLOduration=1.950706163 podStartE2EDuration="1.950706163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:20.950150619 +0000 UTC m=+1720.899731468" watchObservedRunningTime="2026-09-09 16:42:20.950706163 +0000 UTC m=+1720.900287014"
Sep 09 16:42:31.942888 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:31.942857    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:42:35.468597 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.468565    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-6957-8c2ced"]
Sep 09 16:42:35.472091 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.472075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:35.474296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.474275    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-npvh9\""
Sep 09 16:42:35.475165 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.475145    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-8c2ced"]
Sep 09 16:42:35.513485 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.513462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpkh9\" (UniqueName: \"kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9\") pod \"mt-curl-6957-8c2ced\" (UID: \"2c24ece2-4dd0-44ee-982f-ecfe537b47e8\") " pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:35.614431 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.614403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpkh9\" (UniqueName: \"kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9\") pod \"mt-curl-6957-8c2ced\" (UID: \"2c24ece2-4dd0-44ee-982f-ecfe537b47e8\") " pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:35.621894 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.621876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpkh9\" (UniqueName: \"kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9\") pod \"mt-curl-6957-8c2ced\" (UID: \"2c24ece2-4dd0-44ee-982f-ecfe537b47e8\") " pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:35.782249 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:35.782220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:36.051359 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:36.051336    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-8c2ced"]
Sep 09 16:42:36.054005 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:42:36.053983    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c24ece2_4dd0_44ee_982f_ecfe537b47e8.slice/crio-e7d3a3f17c13730df47789c9812a0b2c559990e6bf494825e3036295f8058adf WatchSource:0}: Error finding container e7d3a3f17c13730df47789c9812a0b2c559990e6bf494825e3036295f8058adf: Status 404 returned error can't find the container with id e7d3a3f17c13730df47789c9812a0b2c559990e6bf494825e3036295f8058adf
Sep 09 16:42:37.020030 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:37.019946    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c24ece2-4dd0-44ee-982f-ecfe537b47e8" containerID="c781c82d27160a5972a537464b7b6ecaea135f9e484fd36eeb590a378e5905d1" exitCode=0
Sep 09 16:42:37.020030 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:37.020019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-8c2ced" event={"ID":"2c24ece2-4dd0-44ee-982f-ecfe537b47e8","Type":"ContainerDied","Data":"c781c82d27160a5972a537464b7b6ecaea135f9e484fd36eeb590a378e5905d1"}
Sep 09 16:42:37.020359 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:37.020047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-8c2ced" event={"ID":"2c24ece2-4dd0-44ee-982f-ecfe537b47e8","Type":"ContainerStarted","Data":"e7d3a3f17c13730df47789c9812a0b2c559990e6bf494825e3036295f8058adf"}
Sep 09 16:42:38.149479 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.149453    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:38.184492 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.184470    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-6957-8c2ced_2c24ece2-4dd0-44ee-982f-ecfe537b47e8/mt-curl-6957-8c2ced/0.log"
Sep 09 16:42:38.208187 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.208163    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-6957-8c2ced"]
Sep 09 16:42:38.211293 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.211272    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-6957-8c2ced"]
Sep 09 16:42:38.239023 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.239002    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpkh9\" (UniqueName: \"kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9\") pod \"2c24ece2-4dd0-44ee-982f-ecfe537b47e8\" (UID: \"2c24ece2-4dd0-44ee-982f-ecfe537b47e8\") "
Sep 09 16:42:38.241150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.241125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9" (OuterVolumeSpecName: "kube-api-access-gpkh9") pod "2c24ece2-4dd0-44ee-982f-ecfe537b47e8" (UID: "2c24ece2-4dd0-44ee-982f-ecfe537b47e8"). InnerVolumeSpecName "kube-api-access-gpkh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:42:38.332335 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.332317    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-6957-c65668"]
Sep 09 16:42:38.332699 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.332687    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c24ece2-4dd0-44ee-982f-ecfe537b47e8" containerName="mt-curl-6957-8c2ced"
Sep 09 16:42:38.332737 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.332701    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c24ece2-4dd0-44ee-982f-ecfe537b47e8" containerName="mt-curl-6957-8c2ced"
Sep 09 16:42:38.332791 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.332782    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c24ece2-4dd0-44ee-982f-ecfe537b47e8" containerName="mt-curl-6957-8c2ced"
Sep 09 16:42:38.335637 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.335617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:38.339064 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.339047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-c65668"]
Sep 09 16:42:38.340251 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.340232    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpkh9\" (UniqueName: \"kubernetes.io/projected/2c24ece2-4dd0-44ee-982f-ecfe537b47e8-kube-api-access-gpkh9\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:42:38.440906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.440873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfdm\" (UniqueName: \"kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm\") pod \"mt-curl-6957-c65668\" (UID: \"9601c214-1a75-4075-9a56-c22df9873e1e\") " pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:38.531046 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.531022    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c24ece2-4dd0-44ee-982f-ecfe537b47e8" path="/var/lib/kubelet/pods/2c24ece2-4dd0-44ee-982f-ecfe537b47e8/volumes"
Sep 09 16:42:38.541543 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.541523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfdm\" (UniqueName: \"kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm\") pod \"mt-curl-6957-c65668\" (UID: \"9601c214-1a75-4075-9a56-c22df9873e1e\") " pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:38.548901 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.548880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfdm\" (UniqueName: \"kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm\") pod \"mt-curl-6957-c65668\" (UID: \"9601c214-1a75-4075-9a56-c22df9873e1e\") " pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:38.645170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.645123    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:38.912423 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.912357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-6957-c65668"]
Sep 09 16:42:38.915292 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:42:38.915268    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9601c214_1a75_4075_9a56_c22df9873e1e.slice/crio-4b89b4de79eb1525e31065ac87a8f846547cbd560ffb6f78c8460a847fd2979b WatchSource:0}: Error finding container 4b89b4de79eb1525e31065ac87a8f846547cbd560ffb6f78c8460a847fd2979b: Status 404 returned error can't find the container with id 4b89b4de79eb1525e31065ac87a8f846547cbd560ffb6f78c8460a847fd2979b
Sep 09 16:42:38.916521 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:38.916505    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:42:39.031215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:39.031195    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-8c2ced"
Sep 09 16:42:39.031341 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:39.031266    2559 scope.go:117] "RemoveContainer" containerID="c781c82d27160a5972a537464b7b6ecaea135f9e484fd36eeb590a378e5905d1"
Sep 09 16:42:39.032586 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:39.032563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-c65668" event={"ID":"9601c214-1a75-4075-9a56-c22df9873e1e","Type":"ContainerStarted","Data":"4b89b4de79eb1525e31065ac87a8f846547cbd560ffb6f78c8460a847fd2979b"}
Sep 09 16:42:40.038756 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:40.038724    2559 generic.go:356] "Generic (PLEG): container finished" podID="9601c214-1a75-4075-9a56-c22df9873e1e" containerID="35c4c7735732ef9e54b5570a497d77c8786810361746195a6c923f1f9cd18a04" exitCode=0
Sep 09 16:42:40.039119 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:40.038808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-6957-c65668" event={"ID":"9601c214-1a75-4075-9a56-c22df9873e1e","Type":"ContainerDied","Data":"35c4c7735732ef9e54b5570a497d77c8786810361746195a6c923f1f9cd18a04"}
Sep 09 16:42:41.169320 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.169293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:41.186114 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.186085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-6957-c65668_9601c214-1a75-4075-9a56-c22df9873e1e/mt-curl-6957-c65668/0.log"
Sep 09 16:42:41.205441 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.205409    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-6957-c65668"]
Sep 09 16:42:41.210625 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.210604    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-6957-c65668"]
Sep 09 16:42:41.262708 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.262688    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nfdm\" (UniqueName: \"kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm\") pod \"9601c214-1a75-4075-9a56-c22df9873e1e\" (UID: \"9601c214-1a75-4075-9a56-c22df9873e1e\") "
Sep 09 16:42:41.264633 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.264613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm" (OuterVolumeSpecName: "kube-api-access-5nfdm") pod "9601c214-1a75-4075-9a56-c22df9873e1e" (UID: "9601c214-1a75-4075-9a56-c22df9873e1e"). InnerVolumeSpecName "kube-api-access-5nfdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:42:41.363281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:41.363222    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfdm\" (UniqueName: \"kubernetes.io/projected/9601c214-1a75-4075-9a56-c22df9873e1e-kube-api-access-5nfdm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:42:42.050345 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:42.050316    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:42.050345 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:42.050330    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b89b4de79eb1525e31065ac87a8f846547cbd560ffb6f78c8460a847fd2979b"
Sep 09 16:42:42.060441 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:42:42.060413    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-6957-c65668\" is forbidden: User \"system:node:ip-10-0-133-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-104.ec2.internal' and this object" podUID="9601c214-1a75-4075-9a56-c22df9873e1e" pod="opendatahub/mt-curl-6957-c65668"
Sep 09 16:42:42.537598 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:42.537568    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9601c214-1a75-4075-9a56-c22df9873e1e" path="/var/lib/kubelet/pods/9601c214-1a75-4075-9a56-c22df9873e1e/volumes"
Sep 09 16:42:53.862597 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.862560    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:42:53.863072 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.862975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9601c214-1a75-4075-9a56-c22df9873e1e" containerName="mt-curl-6957-c65668"
Sep 09 16:42:53.863072 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.862998    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9601c214-1a75-4075-9a56-c22df9873e1e" containerName="mt-curl-6957-c65668"
Sep 09 16:42:53.863072 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.863059    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9601c214-1a75-4075-9a56-c22df9873e1e" containerName="mt-curl-6957-c65668"
Sep 09 16:42:53.865915 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.865896    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:53.868334 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.868319    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8cb67b-serving-cert\""
Sep 09 16:42:53.875486 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.875461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:42:53.963874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.963840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:53.964061 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.963903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:53.964061 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.963932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648lg\" (UniqueName: \"kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:53.964061 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:53.964031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.064398 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.064372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.064524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.064418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.064524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.064448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.064524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.064464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648lg\" (UniqueName: \"kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.064697 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:42:54.064584    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-serving-cert: secret "maas-api-e2e-api-b-8cb67b-serving-cert" not found
Sep 09 16:42:54.064697 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:42:54.064648    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls podName:cf7792aa-d0e5-4613-9adf-24247a97c051 nodeName:}" failed. No retries permitted until 2026-09-09 16:42:54.564631533 +0000 UTC m=+1754.514212364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls") pod "maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" (UID: "cf7792aa-d0e5-4613-9adf-24247a97c051") : secret "maas-api-e2e-api-b-8cb67b-serving-cert" not found
Sep 09 16:42:54.065075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.065055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.066824 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.066803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.072100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.072081    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648lg\" (UniqueName: \"kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.484106 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.484076    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:42:54.487680 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.487666    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.490125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.490106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8cb67b\""
Sep 09 16:42:54.490266 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.490199    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8cb67b-dockercfg-hbc8j\""
Sep 09 16:42:54.500378 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.500356    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:42:54.568719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.568693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfmv\" (UniqueName: \"kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.568852 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.568734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.568852 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.568753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.571002 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.570983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") pod \"maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.670099 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.670073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfmv\" (UniqueName: \"kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.670212 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.670109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.670776 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.670757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.679521 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.679499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfmv\" (UniqueName: \"kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv\") pod \"payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") " pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:54.777457 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.777402    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:54.806389 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:54.806359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:55.045916 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:55.045892    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:42:55.048146 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:42:55.048114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7792aa_d0e5_4613_9adf_24247a97c051.slice/crio-59c961487047d1e91484bb575f06cebe8682a2d0dc5e508e2ff7b2cb1beca504 WatchSource:0}: Error finding container 59c961487047d1e91484bb575f06cebe8682a2d0dc5e508e2ff7b2cb1beca504: Status 404 returned error can't find the container with id 59c961487047d1e91484bb575f06cebe8682a2d0dc5e508e2ff7b2cb1beca504
Sep 09 16:42:55.085034 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:55.085012    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:42:55.087150 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:42:55.087119    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be09084_a866_47db_870e_39b10d20e8c4.slice/crio-c93d59f8fb6ad64696d92d54aea0cfcc3ee01f74b19f5838b64adffe68b7c162 WatchSource:0}: Error finding container c93d59f8fb6ad64696d92d54aea0cfcc3ee01f74b19f5838b64adffe68b7c162: Status 404 returned error can't find the container with id c93d59f8fb6ad64696d92d54aea0cfcc3ee01f74b19f5838b64adffe68b7c162
Sep 09 16:42:55.117200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:55.117169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" event={"ID":"2be09084-a866-47db-870e-39b10d20e8c4","Type":"ContainerStarted","Data":"c93d59f8fb6ad64696d92d54aea0cfcc3ee01f74b19f5838b64adffe68b7c162"}
Sep 09 16:42:55.118215 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:55.118195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" event={"ID":"cf7792aa-d0e5-4613-9adf-24247a97c051","Type":"ContainerStarted","Data":"59c961487047d1e91484bb575f06cebe8682a2d0dc5e508e2ff7b2cb1beca504"}
Sep 09 16:42:56.124565 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.124531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" event={"ID":"2be09084-a866-47db-870e-39b10d20e8c4","Type":"ContainerStarted","Data":"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"}
Sep 09 16:42:56.124995 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.124591    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:42:56.126008 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.125987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" event={"ID":"cf7792aa-d0e5-4613-9adf-24247a97c051","Type":"ContainerStarted","Data":"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"}
Sep 09 16:42:56.126110 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.126097    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:42:56.153636 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.153560    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" podStartSLOduration=2.608879979 podStartE2EDuration="3.153544042s" totalImagesPullingTime="544.664063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:56.152888743 +0000 UTC m=+1756.102469612" watchObservedRunningTime="2026-09-09 16:42:56.153544042 +0000 UTC m=+1756.103124889"
Sep 09 16:42:56.154249 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:42:56.154224    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" podStartSLOduration=2.15421795 podStartE2EDuration="2.15421795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:42:56.138661949 +0000 UTC m=+1756.088242793" watchObservedRunningTime="2026-09-09 16:42:56.15421795 +0000 UTC m=+1756.103798796"
Sep 09 16:43:02.136173 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:02.136145    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:43:07.133690 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:07.133658    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:43:39.419520 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.419479    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:43:39.420042 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.419758    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" podUID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" containerName="payload-pre-processing" containerID="cri-o://a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab" gracePeriod=30
Sep 09 16:43:39.664096 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.664072    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:43:39.747979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.747906    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume\") pod \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") "
Sep 09 16:43:39.747979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.747968    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8snth\" (UniqueName: \"kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth\") pod \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\" (UID: \"ce2791db-4a0f-4848-bdf0-ccdefc1010cd\") "
Sep 09 16:43:39.748264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.748244    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce2791db-4a0f-4848-bdf0-ccdefc1010cd" (UID: "ce2791db-4a0f-4848-bdf0-ccdefc1010cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:43:39.750018 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.749986    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth" (OuterVolumeSpecName: "kube-api-access-8snth") pod "ce2791db-4a0f-4848-bdf0-ccdefc1010cd" (UID: "ce2791db-4a0f-4848-bdf0-ccdefc1010cd"). InnerVolumeSpecName "kube-api-access-8snth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:43:39.849491 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.849466    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:43:39.849491 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:39.849487    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8snth\" (UniqueName: \"kubernetes.io/projected/ce2791db-4a0f-4848-bdf0-ccdefc1010cd-kube-api-access-8snth\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:43:40.351708 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.351674    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" containerID="a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab" exitCode=0
Sep 09 16:43:40.351881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.351721    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"
Sep 09 16:43:40.351881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.351749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" event={"ID":"ce2791db-4a0f-4848-bdf0-ccdefc1010cd","Type":"ContainerDied","Data":"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"}
Sep 09 16:43:40.351881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.351803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn" event={"ID":"ce2791db-4a0f-4848-bdf0-ccdefc1010cd","Type":"ContainerDied","Data":"90fcb239580ed7d78bcd943ddbe51a7eb30ce6ee01c08051779b5f60b815a067"}
Sep 09 16:43:40.351881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.351826    2559 scope.go:117] "RemoveContainer" containerID="a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"
Sep 09 16:43:40.360740 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.360726    2559 scope.go:117] "RemoveContainer" containerID="a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"
Sep 09 16:43:40.361009 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:43:40.360991    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab\": container with ID starting with a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab not found: ID does not exist" containerID="a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"
Sep 09 16:43:40.361053 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.361018    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab"} err="failed to get container status \"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab\": rpc error: code = NotFound desc = could not find container \"a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab\": container with ID starting with a51d6b24ebed271a9afc9e7ce070d43c2d4ffd02cda7f5814f6fc78d3b5810ab not found: ID does not exist"
Sep 09 16:43:40.370557 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.370536    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:43:40.376566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.376549    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-14e637-77947d8dfd-jwsbn"]
Sep 09 16:43:40.532237 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.532213    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" path="/var/lib/kubelet/pods/ce2791db-4a0f-4848-bdf0-ccdefc1010cd/volumes"
Sep 09 16:43:40.738102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.738043    2559 scope.go:117] "RemoveContainer" containerID="85e49aa82a6a58aeddac4584a89bf4d93698176a6df835cfcd15c1c846a3e083"
Sep 09 16:43:40.746065 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.746044    2559 scope.go:117] "RemoveContainer" containerID="d50022437691b85b5c18a8f665aa12a8f3544ad17d12748ee6cfefa361652a83"
Sep 09 16:43:40.752954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:40.752929    2559 scope.go:117] "RemoveContainer" containerID="3106c04a35ccdb244327a9cc1c99b677b7db7ef99fc4bf3c6b1779e81767f8bf"
Sep 09 16:43:41.494761 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:41.494734    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:43:41.498157 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:43:41.498134    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:44:01.623945 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.623916    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:44:01.624459 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.624138    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" podUID="cf7792aa-d0e5-4613-9adf-24247a97c051" containerName="maas-api" containerID="cri-o://51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a" gracePeriod=30
Sep 09 16:44:01.673953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.673921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:44:01.674224 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.674194    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" podUID="2be09084-a866-47db-870e-39b10d20e8c4" containerName="payload-processing" containerID="cri-o://8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef" gracePeriod=30
Sep 09 16:44:01.915085 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.915058    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:44:01.943919 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:01.943892    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:44:02.031135 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031100    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-648lg\" (UniqueName: \"kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg\") pod \"cf7792aa-d0e5-4613-9adf-24247a97c051\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") "
Sep 09 16:44:02.031321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031154    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume\") pod \"2be09084-a866-47db-870e-39b10d20e8c4\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") "
Sep 09 16:44:02.031321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031199    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs\") pod \"cf7792aa-d0e5-4613-9adf-24247a97c051\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") "
Sep 09 16:44:02.031321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031227    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca\") pod \"cf7792aa-d0e5-4613-9adf-24247a97c051\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") "
Sep 09 16:44:02.031321 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031316    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") pod \"cf7792aa-d0e5-4613-9adf-24247a97c051\" (UID: \"cf7792aa-d0e5-4613-9adf-24247a97c051\") "
Sep 09 16:44:02.031532 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031364    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmfmv\" (UniqueName: \"kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv\") pod \"2be09084-a866-47db-870e-39b10d20e8c4\" (UID: \"2be09084-a866-47db-870e-39b10d20e8c4\") "
Sep 09 16:44:02.031596 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031568    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2be09084-a866-47db-870e-39b10d20e8c4" (UID: "2be09084-a866-47db-870e-39b10d20e8c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:02.031650 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf7792aa-d0e5-4613-9adf-24247a97c051" (UID: "cf7792aa-d0e5-4613-9adf-24247a97c051"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:02.031707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.031680    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2be09084-a866-47db-870e-39b10d20e8c4-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.033359 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.033335    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg" (OuterVolumeSpecName: "kube-api-access-648lg") pod "cf7792aa-d0e5-4613-9adf-24247a97c051" (UID: "cf7792aa-d0e5-4613-9adf-24247a97c051"). InnerVolumeSpecName "kube-api-access-648lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:02.033464 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.033357    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf7792aa-d0e5-4613-9adf-24247a97c051" (UID: "cf7792aa-d0e5-4613-9adf-24247a97c051"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:02.033464 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.033414    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf7792aa-d0e5-4613-9adf-24247a97c051" (UID: "cf7792aa-d0e5-4613-9adf-24247a97c051"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:02.033537 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.033459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv" (OuterVolumeSpecName: "kube-api-access-qmfmv") pod "2be09084-a866-47db-870e-39b10d20e8c4" (UID: "2be09084-a866-47db-870e-39b10d20e8c4"). InnerVolumeSpecName "kube-api-access-qmfmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:02.132281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.132256    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfmv\" (UniqueName: \"kubernetes.io/projected/2be09084-a866-47db-870e-39b10d20e8c4-kube-api-access-qmfmv\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.132281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.132276    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-648lg\" (UniqueName: \"kubernetes.io/projected/cf7792aa-d0e5-4613-9adf-24247a97c051-kube-api-access-648lg\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.132281 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.132287    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.132467 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.132295    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7792aa-d0e5-4613-9adf-24247a97c051-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.132467 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.132303    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7792aa-d0e5-4613-9adf-24247a97c051-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:02.462240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.462202    2559 generic.go:356] "Generic (PLEG): container finished" podID="cf7792aa-d0e5-4613-9adf-24247a97c051" containerID="51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a" exitCode=0
Sep 09 16:44:02.462433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.462260    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"
Sep 09 16:44:02.462433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.462290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" event={"ID":"cf7792aa-d0e5-4613-9adf-24247a97c051","Type":"ContainerDied","Data":"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"}
Sep 09 16:44:02.462433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.462350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt" event={"ID":"cf7792aa-d0e5-4613-9adf-24247a97c051","Type":"ContainerDied","Data":"59c961487047d1e91484bb575f06cebe8682a2d0dc5e508e2ff7b2cb1beca504"}
Sep 09 16:44:02.462433 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.462378    2559 scope.go:117] "RemoveContainer" containerID="51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"
Sep 09 16:44:02.463751 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.463728    2559 generic.go:356] "Generic (PLEG): container finished" podID="2be09084-a866-47db-870e-39b10d20e8c4" containerID="8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef" exitCode=0
Sep 09 16:44:02.463847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.463760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" event={"ID":"2be09084-a866-47db-870e-39b10d20e8c4","Type":"ContainerDied","Data":"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"}
Sep 09 16:44:02.463847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.463781    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"
Sep 09 16:44:02.463847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.463798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42" event={"ID":"2be09084-a866-47db-870e-39b10d20e8c4","Type":"ContainerDied","Data":"c93d59f8fb6ad64696d92d54aea0cfcc3ee01f74b19f5838b64adffe68b7c162"}
Sep 09 16:44:02.472637 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.472618    2559 scope.go:117] "RemoveContainer" containerID="51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"
Sep 09 16:44:02.473057 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:02.472897    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a\": container with ID starting with 51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a not found: ID does not exist" containerID="51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"
Sep 09 16:44:02.473057 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.472921    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a"} err="failed to get container status \"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a\": rpc error: code = NotFound desc = could not find container \"51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a\": container with ID starting with 51d47603dadea59801282142ca0aea1c42f5ced7b177de85dad1cdaec4f6f72a not found: ID does not exist"
Sep 09 16:44:02.473057 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.472936    2559 scope.go:117] "RemoveContainer" containerID="8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"
Sep 09 16:44:02.480419 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.480404    2559 scope.go:117] "RemoveContainer" containerID="8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"
Sep 09 16:44:02.480665 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:02.480647    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef\": container with ID starting with 8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef not found: ID does not exist" containerID="8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"
Sep 09 16:44:02.480723 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.480672    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef"} err="failed to get container status \"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef\": rpc error: code = NotFound desc = could not find container \"8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef\": container with ID starting with 8de17b90177e29a5008f755ef16c01bf0b22f85e94ddc673cef38e536f5f61ef not found: ID does not exist"
Sep 09 16:44:02.483715 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.483699    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:44:02.486872 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.486850    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8cb67b-684cb6d6f4-ph9vt"]
Sep 09 16:44:02.496443 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.496419    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:44:02.499868 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.499845    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8cb67b-d5778b889-vmr42"]
Sep 09 16:44:02.533322 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.533288    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2be09084-a866-47db-870e-39b10d20e8c4" path="/var/lib/kubelet/pods/2be09084-a866-47db-870e-39b10d20e8c4/volumes"
Sep 09 16:44:02.533670 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:02.533657    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7792aa-d0e5-4613-9adf-24247a97c051" path="/var/lib/kubelet/pods/cf7792aa-d0e5-4613-9adf-24247a97c051/volumes"
Sep 09 16:44:19.546225 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.546164    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:44:19.548655 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.546363    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" podUID="6d89d361-a3da-40bc-9f40-81a9ad657862" containerName="payload-processing" containerID="cri-o://a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727" gracePeriod=30
Sep 09 16:44:19.553769 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.553743    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:44:19.553996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.553954    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" podUID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" containerName="payload-pre-processing" containerID="cri-o://12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8" gracePeriod=30
Sep 09 16:44:19.800073 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.800021    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:44:19.818818 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.818796    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:44:19.980808 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.980780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume\") pod \"6d89d361-a3da-40bc-9f40-81a9ad657862\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") "
Sep 09 16:44:19.980938 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.980823    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume\") pod \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") "
Sep 09 16:44:19.980938 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.980860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm84b\" (UniqueName: \"kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b\") pod \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\" (UID: \"8cb711e8-44e5-4fa0-99f6-d520ba321a84\") "
Sep 09 16:44:19.980938 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.980895    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5d4f\" (UniqueName: \"kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f\") pod \"6d89d361-a3da-40bc-9f40-81a9ad657862\" (UID: \"6d89d361-a3da-40bc-9f40-81a9ad657862\") "
Sep 09 16:44:19.981218 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.981194    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d89d361-a3da-40bc-9f40-81a9ad657862" (UID: "6d89d361-a3da-40bc-9f40-81a9ad657862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:19.981302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.981281    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cb711e8-44e5-4fa0-99f6-d520ba321a84" (UID: "8cb711e8-44e5-4fa0-99f6-d520ba321a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:19.982933 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.982912    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f" (OuterVolumeSpecName: "kube-api-access-m5d4f") pod "6d89d361-a3da-40bc-9f40-81a9ad657862" (UID: "6d89d361-a3da-40bc-9f40-81a9ad657862"). InnerVolumeSpecName "kube-api-access-m5d4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:19.983020 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:19.982932    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b" (OuterVolumeSpecName: "kube-api-access-pm84b") pod "8cb711e8-44e5-4fa0-99f6-d520ba321a84" (UID: "8cb711e8-44e5-4fa0-99f6-d520ba321a84"). InnerVolumeSpecName "kube-api-access-pm84b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:20.081994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.081917    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5d4f\" (UniqueName: \"kubernetes.io/projected/6d89d361-a3da-40bc-9f40-81a9ad657862-kube-api-access-m5d4f\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:20.081994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.081939    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d89d361-a3da-40bc-9f40-81a9ad657862-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:20.081994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.081947    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cb711e8-44e5-4fa0-99f6-d520ba321a84-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:20.081994 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.081955    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm84b\" (UniqueName: \"kubernetes.io/projected/8cb711e8-44e5-4fa0-99f6-d520ba321a84-kube-api-access-pm84b\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:20.566339 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.566316    2559 generic.go:356] "Generic (PLEG): container finished" podID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" containerID="12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8" exitCode=0
Sep 09 16:44:20.566638 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.566362    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"
Sep 09 16:44:20.566638 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.566400    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" event={"ID":"8cb711e8-44e5-4fa0-99f6-d520ba321a84","Type":"ContainerDied","Data":"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"}
Sep 09 16:44:20.566638 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.566455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k" event={"ID":"8cb711e8-44e5-4fa0-99f6-d520ba321a84","Type":"ContainerDied","Data":"a4c90fde96b84d82ce4f17994638b9caf1751e3633fadfe794838e3502cfde15"}
Sep 09 16:44:20.566638 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.566485    2559 scope.go:117] "RemoveContainer" containerID="12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"
Sep 09 16:44:20.567807 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.567771    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d89d361-a3da-40bc-9f40-81a9ad657862" containerID="a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727" exitCode=0
Sep 09 16:44:20.567918 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.567831    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"
Sep 09 16:44:20.567918 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.567852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" event={"ID":"6d89d361-a3da-40bc-9f40-81a9ad657862","Type":"ContainerDied","Data":"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"}
Sep 09 16:44:20.567918 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.567889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr" event={"ID":"6d89d361-a3da-40bc-9f40-81a9ad657862","Type":"ContainerDied","Data":"85b61ca5a5e67044b93c51b0ddb5672e18aa58a20e772685fde4f1b933342132"}
Sep 09 16:44:20.574685 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.574662    2559 scope.go:117] "RemoveContainer" containerID="12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"
Sep 09 16:44:20.574931 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:20.574913    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8\": container with ID starting with 12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8 not found: ID does not exist" containerID="12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"
Sep 09 16:44:20.575009 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.574939    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8"} err="failed to get container status \"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8\": rpc error: code = NotFound desc = could not find container \"12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8\": container with ID starting with 12dba74ab72b993d7fac49a715b76a9a9da84f1ae64c578ed39d3460e2a09ec8 not found: ID does not exist"
Sep 09 16:44:20.575009 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.574973    2559 scope.go:117] "RemoveContainer" containerID="a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"
Sep 09 16:44:20.582671 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.582660    2559 scope.go:117] "RemoveContainer" containerID="a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"
Sep 09 16:44:20.582988 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:20.582936    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727\": container with ID starting with a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727 not found: ID does not exist" containerID="a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"
Sep 09 16:44:20.583171 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.582988    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727"} err="failed to get container status \"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727\": rpc error: code = NotFound desc = could not find container \"a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727\": container with ID starting with a3f27591cf7eb145819e961633a4bc4403df9b4eb3a8ec5593856a7a9fdf6727 not found: ID does not exist"
Sep 09 16:44:20.585194 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.585176    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:44:20.588828 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.588808    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bdf45-74bcc4f7f-567sr"]
Sep 09 16:44:20.599142 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.599121    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:44:20.602524 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:20.602505    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bdf45-646dbc9f9d-l7q4k"]
Sep 09 16:44:22.530991 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:22.530942    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d89d361-a3da-40bc-9f40-81a9ad657862" path="/var/lib/kubelet/pods/6d89d361-a3da-40bc-9f40-81a9ad657862/volumes"
Sep 09 16:44:22.531349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:22.531283    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" path="/var/lib/kubelet/pods/8cb711e8-44e5-4fa0-99f6-d520ba321a84/volumes"
Sep 09 16:44:36.804284 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:36.804249    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:44:36.804819 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:36.804520    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" podUID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" containerName="maas-api" containerID="cri-o://db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97" gracePeriod=30
Sep 09 16:44:36.852121 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:36.852092    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:44:36.852355 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:36.852331    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" podUID="c673fe08-f9e5-419f-8abf-42968ac7c39b" containerName="payload-processing" containerID="cri-o://53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4" gracePeriod=30
Sep 09 16:44:37.090280 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.090255    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:44:37.119070 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.119044    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:44:37.123205 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123182    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca\") pod \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") "
Sep 09 16:44:37.123308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") pod \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") "
Sep 09 16:44:37.123308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs\") pod \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") "
Sep 09 16:44:37.123397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqd47\" (UniqueName: \"kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47\") pod \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\" (UID: \"9b2c3d7c-7a7c-4dde-a931-13dccd71b090\") "
Sep 09 16:44:37.123552 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123526    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b2c3d7c-7a7c-4dde-a931-13dccd71b090" (UID: "9b2c3d7c-7a7c-4dde-a931-13dccd71b090"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:37.123687 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.123670    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.125372 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.125351    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47" (OuterVolumeSpecName: "kube-api-access-cqd47") pod "9b2c3d7c-7a7c-4dde-a931-13dccd71b090" (UID: "9b2c3d7c-7a7c-4dde-a931-13dccd71b090"). InnerVolumeSpecName "kube-api-access-cqd47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:37.125739 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.125714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b2c3d7c-7a7c-4dde-a931-13dccd71b090" (UID: "9b2c3d7c-7a7c-4dde-a931-13dccd71b090"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:37.125739 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.125734    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b2c3d7c-7a7c-4dde-a931-13dccd71b090" (UID: "9b2c3d7c-7a7c-4dde-a931-13dccd71b090"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:44:37.224456 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224434    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85wgw\" (UniqueName: \"kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw\") pod \"c673fe08-f9e5-419f-8abf-42968ac7c39b\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") "
Sep 09 16:44:37.224584 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume\") pod \"c673fe08-f9e5-419f-8abf-42968ac7c39b\" (UID: \"c673fe08-f9e5-419f-8abf-42968ac7c39b\") "
Sep 09 16:44:37.224696 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224680    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.224759 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224701    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.224759 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224714    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqd47\" (UniqueName: \"kubernetes.io/projected/9b2c3d7c-7a7c-4dde-a931-13dccd71b090-kube-api-access-cqd47\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.224874 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.224859    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c673fe08-f9e5-419f-8abf-42968ac7c39b" (UID: "c673fe08-f9e5-419f-8abf-42968ac7c39b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:44:37.226255 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.226233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw" (OuterVolumeSpecName: "kube-api-access-85wgw") pod "c673fe08-f9e5-419f-8abf-42968ac7c39b" (UID: "c673fe08-f9e5-419f-8abf-42968ac7c39b"). InnerVolumeSpecName "kube-api-access-85wgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:44:37.325702 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.325642    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c673fe08-f9e5-419f-8abf-42968ac7c39b-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.325702 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.325667    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wgw\" (UniqueName: \"kubernetes.io/projected/c673fe08-f9e5-419f-8abf-42968ac7c39b-kube-api-access-85wgw\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:44:37.650537 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.650467    2559 generic.go:356] "Generic (PLEG): container finished" podID="c673fe08-f9e5-419f-8abf-42968ac7c39b" containerID="53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4" exitCode=0
Sep 09 16:44:37.650537 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.650511    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"
Sep 09 16:44:37.650726 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.650561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" event={"ID":"c673fe08-f9e5-419f-8abf-42968ac7c39b","Type":"ContainerDied","Data":"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"}
Sep 09 16:44:37.650726 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.650616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk" event={"ID":"c673fe08-f9e5-419f-8abf-42968ac7c39b","Type":"ContainerDied","Data":"35418d9919e11b258f58ed84c99831e727622cccf657f068e1b164790aded453"}
Sep 09 16:44:37.650726 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.650647    2559 scope.go:117] "RemoveContainer" containerID="53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"
Sep 09 16:44:37.652582 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.652556    2559 generic.go:356] "Generic (PLEG): container finished" podID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" containerID="db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97" exitCode=0
Sep 09 16:44:37.652700 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.652597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" event={"ID":"9b2c3d7c-7a7c-4dde-a931-13dccd71b090","Type":"ContainerDied","Data":"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"}
Sep 09 16:44:37.652700 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.652604    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"
Sep 09 16:44:37.652700 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.652636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5" event={"ID":"9b2c3d7c-7a7c-4dde-a931-13dccd71b090","Type":"ContainerDied","Data":"8e6801affb25f06b6205d45c1dedafd04e64982f78afbba76866592233274bef"}
Sep 09 16:44:37.659554 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.659535    2559 scope.go:117] "RemoveContainer" containerID="53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"
Sep 09 16:44:37.659840 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:37.659815    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4\": container with ID starting with 53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4 not found: ID does not exist" containerID="53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"
Sep 09 16:44:37.659928 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.659853    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4"} err="failed to get container status \"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4\": rpc error: code = NotFound desc = could not find container \"53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4\": container with ID starting with 53caf888c9886e618d133beecbb0625c73d942bae683cd859397f3dbabee65d4 not found: ID does not exist"
Sep 09 16:44:37.659928 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.659875    2559 scope.go:117] "RemoveContainer" containerID="db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"
Sep 09 16:44:37.667530 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.667514    2559 scope.go:117] "RemoveContainer" containerID="db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"
Sep 09 16:44:37.667752 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:37.667734    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97\": container with ID starting with db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97 not found: ID does not exist" containerID="db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"
Sep 09 16:44:37.667814 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.667762    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97"} err="failed to get container status \"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97\": rpc error: code = NotFound desc = could not find container \"db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97\": container with ID starting with db200a6e70755a4946cab01b352970199d170aebd9879b62ac3368d9e04cbf97 not found: ID does not exist"
Sep 09 16:44:37.675671 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.675652    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:44:37.678990 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.678972    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-1d59dd-9db848c84-58gx5"]
Sep 09 16:44:37.690844 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.690825    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:44:37.694765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:37.694744    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1d59dd-65b4b5f8fc-jccnk"]
Sep 09 16:44:38.530788 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:38.530755    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" path="/var/lib/kubelet/pods/9b2c3d7c-7a7c-4dde-a931-13dccd71b090/volumes"
Sep 09 16:44:38.531141 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:38.531109    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c673fe08-f9e5-419f-8abf-42968ac7c39b" path="/var/lib/kubelet/pods/c673fe08-f9e5-419f-8abf-42968ac7c39b/volumes"
Sep 09 16:44:41.139837 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.139753    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140101    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140113    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140120    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2be09084-a866-47db-870e-39b10d20e8c4" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140127    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2be09084-a866-47db-870e-39b10d20e8c4" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140136    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c673fe08-f9e5-419f-8abf-42968ac7c39b" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140142    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c673fe08-f9e5-419f-8abf-42968ac7c39b" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140154    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7792aa-d0e5-4613-9adf-24247a97c051" containerName="maas-api"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140159    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7792aa-d0e5-4613-9adf-24247a97c051" containerName="maas-api"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140167    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d89d361-a3da-40bc-9f40-81a9ad657862" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140172    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d89d361-a3da-40bc-9f40-81a9ad657862" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140180    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140185    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140192    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" containerName="maas-api"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140197    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" containerName="maas-api"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140250    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d89d361-a3da-40bc-9f40-81a9ad657862" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140258    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2791db-4a0f-4848-bdf0-ccdefc1010cd" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140264    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2be09084-a866-47db-870e-39b10d20e8c4" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140270    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7792aa-d0e5-4613-9adf-24247a97c051" containerName="maas-api"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140276    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb711e8-44e5-4fa0-99f6-d520ba321a84" containerName="payload-pre-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c673fe08-f9e5-419f-8abf-42968ac7c39b" containerName="payload-processing"
Sep 09 16:44:41.140279 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.140288    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2c3d7c-7a7c-4dde-a931-13dccd71b090" containerName="maas-api"
Sep 09 16:44:41.144947 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.144928    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.147230 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.147213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-61f0f9-serving-cert\""
Sep 09 16:44:41.152816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.152361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:44:41.257137 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.257100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.257259 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.257216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.257307 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.257259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.257353 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.257330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5tn\" (UniqueName: \"kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.358102 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.358068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5tn\" (UniqueName: \"kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.358212 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.358118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.358212 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.358169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.358328 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.358212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.358328 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:41.358297    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-serving-cert: secret "maas-api-e2e-ipp-a-61f0f9-serving-cert" not found
Sep 09 16:44:41.358426 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:41.358366    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls podName:938b19e7-71ce-4dd9-85fa-61bb50f2df71 nodeName:}" failed. No retries permitted until 2026-09-09 16:44:41.858344834 +0000 UTC m=+1861.807925660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls") pod "maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71") : secret "maas-api-e2e-ipp-a-61f0f9-serving-cert" not found
Sep 09 16:44:41.358736 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.358718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.360612 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.360592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.366218 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.366202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5tn\" (UniqueName: \"kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.861518 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:41.861488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:41.861671 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:41.861596    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-serving-cert: secret "maas-api-e2e-ipp-a-61f0f9-serving-cert" not found
Sep 09 16:44:41.861671 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:44:41.861650    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls podName:938b19e7-71ce-4dd9-85fa-61bb50f2df71 nodeName:}" failed. No retries permitted until 2026-09-09 16:44:42.861632236 +0000 UTC m=+1862.811213063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls") pod "maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71") : secret "maas-api-e2e-ipp-a-61f0f9-serving-cert" not found
Sep 09 16:44:42.867442 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:42.867408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:42.869659 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:42.869638    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:42.957476 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:42.957453    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:43.226844 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:43.226776    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:44:43.230220 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:44:43.230194    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938b19e7_71ce_4dd9_85fa_61bb50f2df71.slice/crio-bc6ac15e1e28e7e632c26280744cd88ec18b89eefb2fdadb29622c1e15a63632 WatchSource:0}: Error finding container bc6ac15e1e28e7e632c26280744cd88ec18b89eefb2fdadb29622c1e15a63632: Status 404 returned error can't find the container with id bc6ac15e1e28e7e632c26280744cd88ec18b89eefb2fdadb29622c1e15a63632
Sep 09 16:44:43.682763 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:43.682724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" event={"ID":"938b19e7-71ce-4dd9-85fa-61bb50f2df71","Type":"ContainerStarted","Data":"bc6ac15e1e28e7e632c26280744cd88ec18b89eefb2fdadb29622c1e15a63632"}
Sep 09 16:44:44.689686 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:44.689661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" event={"ID":"938b19e7-71ce-4dd9-85fa-61bb50f2df71","Type":"ContainerStarted","Data":"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"}
Sep 09 16:44:44.690040 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:44.689795    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:44:44.706996 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:44.706933    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" podStartSLOduration=2.307889063 podStartE2EDuration="3.706914945s" totalImagesPullingTime="1.399025882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:44:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:44:44.705490966 +0000 UTC m=+1864.655071815" watchObservedRunningTime="2026-09-09 16:44:44.706914945 +0000 UTC m=+1864.656495791"
Sep 09 16:44:50.699075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:44:50.699045    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:45:10.930011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:10.929909    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:45:10.933461 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:10.933443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:10.935847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:10.935825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cd0c91-dockercfg-r2bqv\""
Sep 09 16:45:10.935934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:10.935860    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cd0c91\""
Sep 09 16:45:10.942129 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:10.942104    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:45:11.087434 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.087398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzfq\" (UniqueName: \"kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.087606 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.087458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.188232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.188142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzfq\" (UniqueName: \"kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.188232 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.188218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.188854 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.188835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.195234 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.195214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzfq\" (UniqueName: \"kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq\") pod \"payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.244741 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.244718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.530927 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.530901    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:45:11.532374 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:45:11.532351    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f7c24f_c704_4412_bcff_85b560db9125.slice/crio-0368511941ded0900fb77e9e4681670a1f7a1f43d74726afe145f0cb2770bdb8 WatchSource:0}: Error finding container 0368511941ded0900fb77e9e4681670a1f7a1f43d74726afe145f0cb2770bdb8: Status 404 returned error can't find the container with id 0368511941ded0900fb77e9e4681670a1f7a1f43d74726afe145f0cb2770bdb8
Sep 09 16:45:11.810948 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.810875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" event={"ID":"c0f7c24f-c704-4412-bcff-85b560db9125","Type":"ContainerStarted","Data":"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"}
Sep 09 16:45:11.810948 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.810921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" event={"ID":"c0f7c24f-c704-4412-bcff-85b560db9125","Type":"ContainerStarted","Data":"0368511941ded0900fb77e9e4681670a1f7a1f43d74726afe145f0cb2770bdb8"}
Sep 09 16:45:11.811124 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.810976    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:11.827043 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:11.827003    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" podStartSLOduration=1.826990141 podStartE2EDuration="1.826990141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:45:11.825482277 +0000 UTC m=+1891.775063123" watchObservedRunningTime="2026-09-09 16:45:11.826990141 +0000 UTC m=+1891.776571002"
Sep 09 16:45:22.816975 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:22.816939    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:45:48.399520 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.399484    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:45:48.402891 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.402869    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.406193 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.406174    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-61f0f9\"/\"kube-root-ca.crt\""
Sep 09 16:45:48.406308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.406192    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-61f0f9\"/\"ipp-route-61f0f9-kserve-self-signed-certs\""
Sep 09 16:45:48.406308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.406203    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-61f0f9\"/\"default-dockercfg-wxxmk\""
Sep 09 16:45:48.406308 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.406194    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-61f0f9\"/\"openshift-service-ca.crt\""
Sep 09 16:45:48.413819 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.413801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:45:48.456725 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.456816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.456816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.456890 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmtz\" (UniqueName: \"kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.456925 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.456925 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.456917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557396 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557370    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmtz\" (UniqueName: \"kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557478 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557478 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557575 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557575 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557575 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.557890 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.557825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.558100 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.558080    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.559707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.559686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.559948 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.559934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.565558 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.565535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmtz\" (UniqueName: \"kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz\") pod \"ipp-route-61f0f9-kserve-8676566d6c-4b4n2\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") " pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.711877 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.711824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:48.978201 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:48.978134    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:45:48.981608 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:45:48.981580    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8ffefb0_57fb_45de_9607_32a9b2df33c6.slice/crio-88d7236945381b24f8a7162fc60ea86d90bfdbeb18a78c7f3794c0c08b3e537e WatchSource:0}: Error finding container 88d7236945381b24f8a7162fc60ea86d90bfdbeb18a78c7f3794c0c08b3e537e: Status 404 returned error can't find the container with id 88d7236945381b24f8a7162fc60ea86d90bfdbeb18a78c7f3794c0c08b3e537e
Sep 09 16:45:49.987077 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:49.987044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" event={"ID":"d8ffefb0-57fb-45de-9607-32a9b2df33c6","Type":"ContainerStarted","Data":"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"}
Sep 09 16:45:49.987077 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:49.987081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" event={"ID":"d8ffefb0-57fb-45de-9607-32a9b2df33c6","Type":"ContainerStarted","Data":"88d7236945381b24f8a7162fc60ea86d90bfdbeb18a78c7f3794c0c08b3e537e"}
Sep 09 16:45:50.004278 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:50.004231    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" podStartSLOduration=2.004216711 podStartE2EDuration="2.004216711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:45:50.001489134 +0000 UTC m=+1929.951069981" watchObservedRunningTime="2026-09-09 16:45:50.004216711 +0000 UTC m=+1929.953797557"
Sep 09 16:45:58.711973 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:58.711928    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:58.712348 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:58.711986    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:58.723878 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:58.723853    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:45:59.044038 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:45:59.044014    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:46:31.954775 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:31.954743    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:46:31.958711 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:31.958691    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:31.960875 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:31.960857    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-89cc2b\""
Sep 09 16:46:31.961011 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:31.960989    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-89cc2b-dockercfg-jc4s7\""
Sep 09 16:46:31.967531 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:31.967512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:46:32.076743 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.076721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwpb\" (UniqueName: \"kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.076847 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.076762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.177906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.177887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwpb\" (UniqueName: \"kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.178016 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.177926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.178547 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.178530    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.185104 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.185079    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwpb\" (UniqueName: \"kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb\") pod \"payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.269628 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.269583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:32.568645 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:32.568623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:46:32.570926 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:46:32.570893    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c1a7ec_75ee_4c1a_bc98_c1ec5c11ab14.slice/crio-5b7d4bb29110f7c5eb6d2d7347112901d88c754eb8d01e805b9ca01982488ee8 WatchSource:0}: Error finding container 5b7d4bb29110f7c5eb6d2d7347112901d88c754eb8d01e805b9ca01982488ee8: Status 404 returned error can't find the container with id 5b7d4bb29110f7c5eb6d2d7347112901d88c754eb8d01e805b9ca01982488ee8
Sep 09 16:46:33.192985 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:33.192907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" event={"ID":"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14","Type":"ContainerStarted","Data":"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"}
Sep 09 16:46:33.193343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:33.192985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" event={"ID":"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14","Type":"ContainerStarted","Data":"5b7d4bb29110f7c5eb6d2d7347112901d88c754eb8d01e805b9ca01982488ee8"}
Sep 09 16:46:33.193343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:33.193051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:46:33.206170 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:33.206120    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" podStartSLOduration=2.206108545 podStartE2EDuration="2.206108545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:46:33.204528075 +0000 UTC m=+1973.154108922" watchObservedRunningTime="2026-09-09 16:46:33.206108545 +0000 UTC m=+1973.155689390"
Sep 09 16:46:44.199428 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:46:44.199398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:47:10.480526 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.480319    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:47:10.481085 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.480654    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" podUID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" containerName="payload-pre-processing" containerID="cri-o://8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b" gracePeriod=30
Sep 09 16:47:10.728999 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.728979    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:47:10.784133 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.784109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcwpb\" (UniqueName: \"kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb\") pod \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") "
Sep 09 16:47:10.784264 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.784181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume\") pod \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\" (UID: \"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14\") "
Sep 09 16:47:10.784544 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.784521    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" (UID: "e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:10.786247 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.786220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb" (OuterVolumeSpecName: "kube-api-access-fcwpb") pod "e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" (UID: "e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14"). InnerVolumeSpecName "kube-api-access-fcwpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:10.885136 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.885112    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:10.885136 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:10.885134    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwpb\" (UniqueName: \"kubernetes.io/projected/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14-kube-api-access-fcwpb\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:11.379066 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.379037    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" containerID="8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b" exitCode=0
Sep 09 16:47:11.379244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.379085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" event={"ID":"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14","Type":"ContainerDied","Data":"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"}
Sep 09 16:47:11.379244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.379089    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"
Sep 09 16:47:11.379244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.379122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr" event={"ID":"e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14","Type":"ContainerDied","Data":"5b7d4bb29110f7c5eb6d2d7347112901d88c754eb8d01e805b9ca01982488ee8"}
Sep 09 16:47:11.379244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.379145    2559 scope.go:117] "RemoveContainer" containerID="8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"
Sep 09 16:47:11.387816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.387797    2559 scope.go:117] "RemoveContainer" containerID="8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"
Sep 09 16:47:11.388105 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:47:11.388078    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b\": container with ID starting with 8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b not found: ID does not exist" containerID="8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"
Sep 09 16:47:11.388172 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.388105    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b"} err="failed to get container status \"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b\": rpc error: code = NotFound desc = could not find container \"8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b\": container with ID starting with 8b0cf5007f1c038eaf3b4752bbc1f314c7485106a3a5926f1b20fca36e613e7b not found: ID does not exist"
Sep 09 16:47:11.398448 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.398426    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:47:11.404974 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:11.404937    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-89cc2b-68896bd866-f8jvr"]
Sep 09 16:47:12.531505 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:12.531473    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" path="/var/lib/kubelet/pods/e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14/volumes"
Sep 09 16:47:36.075083 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.075047    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:47:36.075498 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.075359    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" podUID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" containerName="maas-api" containerID="cri-o://88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236" gracePeriod=30
Sep 09 16:47:36.318207 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.318182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:47:36.391303 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.391218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs\") pod \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") "
Sep 09 16:47:36.391303 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.391278    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") pod \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") "
Sep 09 16:47:36.391303 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.391307    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp5tn\" (UniqueName: \"kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn\") pod \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") "
Sep 09 16:47:36.391581 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.391367    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca\") pod \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\" (UID: \"938b19e7-71ce-4dd9-85fa-61bb50f2df71\") "
Sep 09 16:47:36.391754 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.391726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "938b19e7-71ce-4dd9-85fa-61bb50f2df71" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:47:36.393297 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.393272    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "938b19e7-71ce-4dd9-85fa-61bb50f2df71" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:47:36.393409 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.393326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "938b19e7-71ce-4dd9-85fa-61bb50f2df71" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:47:36.393409 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.393345    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn" (OuterVolumeSpecName: "kube-api-access-rp5tn") pod "938b19e7-71ce-4dd9-85fa-61bb50f2df71" (UID: "938b19e7-71ce-4dd9-85fa-61bb50f2df71"). InnerVolumeSpecName "kube-api-access-rp5tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:36.492190 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.492161    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.492190 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.492184    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/938b19e7-71ce-4dd9-85fa-61bb50f2df71-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.492190 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.492195    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5tn\" (UniqueName: \"kubernetes.io/projected/938b19e7-71ce-4dd9-85fa-61bb50f2df71-kube-api-access-rp5tn\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.492402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.492204    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/938b19e7-71ce-4dd9-85fa-61bb50f2df71-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:36.497229 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.497204    2559 generic.go:356] "Generic (PLEG): container finished" podID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" containerID="88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236" exitCode=0
Sep 09 16:47:36.497333 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.497245    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"
Sep 09 16:47:36.497372 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.497246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" event={"ID":"938b19e7-71ce-4dd9-85fa-61bb50f2df71","Type":"ContainerDied","Data":"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"}
Sep 09 16:47:36.497407 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.497378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql" event={"ID":"938b19e7-71ce-4dd9-85fa-61bb50f2df71","Type":"ContainerDied","Data":"bc6ac15e1e28e7e632c26280744cd88ec18b89eefb2fdadb29622c1e15a63632"}
Sep 09 16:47:36.497440 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.497406    2559 scope.go:117] "RemoveContainer" containerID="88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"
Sep 09 16:47:36.507856 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.507840    2559 scope.go:117] "RemoveContainer" containerID="88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"
Sep 09 16:47:36.509043 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:47:36.509023    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236\": container with ID starting with 88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236 not found: ID does not exist" containerID="88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"
Sep 09 16:47:36.509125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.509049    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236"} err="failed to get container status \"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236\": rpc error: code = NotFound desc = could not find container \"88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236\": container with ID starting with 88f4b7027bc7ab553e72739ff3e0170f6671c88516fb6b1db46fde627aef3236 not found: ID does not exist"
Sep 09 16:47:36.517048 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.517028    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:47:36.520197 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.520178    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-61f0f9-5857fcc558-h9dql"]
Sep 09 16:47:36.532209 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:36.532179    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" path="/var/lib/kubelet/pods/938b19e7-71ce-4dd9-85fa-61bb50f2df71/volumes"
Sep 09 16:47:41.671219 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:41.671179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:47:56.689995 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:56.689937    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" podUID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" containerName="main" containerID="cri-o://4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c" gracePeriod=45
Sep 09 16:47:56.940648 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:56.940596    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:47:57.040402 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040376    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040534 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040534 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040534 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040467    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhmtz\" (UniqueName: \"kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040691 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040691 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040611    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm\") pod \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\" (UID: \"d8ffefb0-57fb-45de-9607-32a9b2df33c6\") "
Sep 09 16:47:57.040691 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040670    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:47:57.040691 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040676    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:47:57.040896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040692    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home" (OuterVolumeSpecName: "home") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:47:57.040896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040847    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-home\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.040896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040858    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tmp-dir\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.040896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.040869    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-model-cache\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.042590 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.042572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm" (OuterVolumeSpecName: "dshm") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 16:47:57.042697 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.042673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:47:57.042782 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.042758    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz" (OuterVolumeSpecName: "kube-api-access-rhmtz") pod "d8ffefb0-57fb-45de-9607-32a9b2df33c6" (UID: "d8ffefb0-57fb-45de-9607-32a9b2df33c6"). InnerVolumeSpecName "kube-api-access-rhmtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:47:57.142240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.142217    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmtz\" (UniqueName: \"kubernetes.io/projected/d8ffefb0-57fb-45de-9607-32a9b2df33c6-kube-api-access-rhmtz\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.142240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.142237    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ffefb0-57fb-45de-9607-32a9b2df33c6-tls-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.142368 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.142247    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8ffefb0-57fb-45de-9607-32a9b2df33c6-dshm\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:47:57.592450 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.592427    2559 generic.go:356] "Generic (PLEG): container finished" podID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" containerID="4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c" exitCode=0
Sep 09 16:47:57.592588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.592465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" event={"ID":"d8ffefb0-57fb-45de-9607-32a9b2df33c6","Type":"ContainerDied","Data":"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"}
Sep 09 16:47:57.592588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.592473    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"
Sep 09 16:47:57.592588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.592487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2" event={"ID":"d8ffefb0-57fb-45de-9607-32a9b2df33c6","Type":"ContainerDied","Data":"88d7236945381b24f8a7162fc60ea86d90bfdbeb18a78c7f3794c0c08b3e537e"}
Sep 09 16:47:57.592588 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.592504    2559 scope.go:117] "RemoveContainer" containerID="4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"
Sep 09 16:47:57.600607 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.600592    2559 scope.go:117] "RemoveContainer" containerID="4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"
Sep 09 16:47:57.600861 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:47:57.600844    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c\": container with ID starting with 4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c not found: ID does not exist" containerID="4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"
Sep 09 16:47:57.600907 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.600869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c"} err="failed to get container status \"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c\": rpc error: code = NotFound desc = could not find container \"4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c\": container with ID starting with 4f57994c3e6f9f77e5819aeae9a2076dcb757a2cebdd115972dcfe6cf2c3a29c not found: ID does not exist"
Sep 09 16:47:57.610824 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.610804    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:47:57.614608 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:57.614589    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-61f0f9/ipp-route-61f0f9-kserve-8676566d6c-4b4n2"]
Sep 09 16:47:58.531238 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:47:58.531197    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" path="/var/lib/kubelet/pods/d8ffefb0-57fb-45de-9607-32a9b2df33c6/volumes"
Sep 09 16:48:35.384125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.384092    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:48:35.384718 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.384341    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" podUID="c0f7c24f-c704-4412-bcff-85b560db9125" containerName="payload-pre-processing" containerID="cri-o://72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6" gracePeriod=30
Sep 09 16:48:35.625751 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.625728    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:48:35.651174 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.651084    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxzfq\" (UniqueName: \"kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq\") pod \"c0f7c24f-c704-4412-bcff-85b560db9125\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") "
Sep 09 16:48:35.651174 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.651129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume\") pod \"c0f7c24f-c704-4412-bcff-85b560db9125\" (UID: \"c0f7c24f-c704-4412-bcff-85b560db9125\") "
Sep 09 16:48:35.651836 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.651809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0f7c24f-c704-4412-bcff-85b560db9125" (UID: "c0f7c24f-c704-4412-bcff-85b560db9125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:35.653566 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.653539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq" (OuterVolumeSpecName: "kube-api-access-mxzfq") pod "c0f7c24f-c704-4412-bcff-85b560db9125" (UID: "c0f7c24f-c704-4412-bcff-85b560db9125"). InnerVolumeSpecName "kube-api-access-mxzfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:35.753040 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.753014    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzfq\" (UniqueName: \"kubernetes.io/projected/c0f7c24f-c704-4412-bcff-85b560db9125-kube-api-access-mxzfq\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.753040 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.753038    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f7c24f-c704-4412-bcff-85b560db9125-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:35.764950 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.764920    2559 generic.go:356] "Generic (PLEG): container finished" podID="c0f7c24f-c704-4412-bcff-85b560db9125" containerID="72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6" exitCode=0
Sep 09 16:48:35.765081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.764974    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"
Sep 09 16:48:35.765081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.764995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" event={"ID":"c0f7c24f-c704-4412-bcff-85b560db9125","Type":"ContainerDied","Data":"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"}
Sep 09 16:48:35.765081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.765046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx" event={"ID":"c0f7c24f-c704-4412-bcff-85b560db9125","Type":"ContainerDied","Data":"0368511941ded0900fb77e9e4681670a1f7a1f43d74726afe145f0cb2770bdb8"}
Sep 09 16:48:35.765081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.765078    2559 scope.go:117] "RemoveContainer" containerID="72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"
Sep 09 16:48:35.773142 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.773126    2559 scope.go:117] "RemoveContainer" containerID="72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"
Sep 09 16:48:35.773383 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:48:35.773366    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6\": container with ID starting with 72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6 not found: ID does not exist" containerID="72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"
Sep 09 16:48:35.773430 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.773390    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6"} err="failed to get container status \"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6\": rpc error: code = NotFound desc = could not find container \"72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6\": container with ID starting with 72f7ded271d336630f561836fb2b54a52b384496aff2cb12d0c549cac3d32ff6 not found: ID does not exist"
Sep 09 16:48:35.783921 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.783898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:48:35.785541 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:35.785523    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cd0c91-664bfb8466-2xnrx"]
Sep 09 16:48:36.530750 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:36.530719    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f7c24f-c704-4412-bcff-85b560db9125" path="/var/lib/kubelet/pods/c0f7c24f-c704-4412-bcff-85b560db9125/volumes"
Sep 09 16:48:40.835442 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:40.835412    2559 scope.go:117] "RemoveContainer" containerID="35c4c7735732ef9e54b5570a497d77c8786810361746195a6c923f1f9cd18a04"
Sep 09 16:48:41.523315 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:41.523292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:48:41.528285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:41.528268    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:48:52.838032 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:52.837989    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:48:52.838509 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:52.838202    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" podUID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" containerName="maas-api" containerID="cri-o://1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305" gracePeriod=30
Sep 09 16:48:52.894589 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:52.894554    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:48:52.894836 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:52.894812    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" podUID="3520d030-07ef-470a-b624-7f52dd765be3" containerName="payload-pre-processing" containerID="cri-o://f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6" gracePeriod=30
Sep 09 16:48:53.141830 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.141805    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:48:53.145684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.145667    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:48:53.279250 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279223    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume\") pod \"3520d030-07ef-470a-b624-7f52dd765be3\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") "
Sep 09 16:48:53.279411 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279319    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca\") pod \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") "
Sep 09 16:48:53.279411 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279356    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mks4m\" (UniqueName: \"kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m\") pod \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") "
Sep 09 16:48:53.279411 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279383    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-492qj\" (UniqueName: \"kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj\") pod \"3520d030-07ef-470a-b624-7f52dd765be3\" (UID: \"3520d030-07ef-470a-b624-7f52dd765be3\") "
Sep 09 16:48:53.279596 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279414    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs\") pod \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") "
Sep 09 16:48:53.279596 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls\") pod \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\" (UID: \"ba89743d-3a8d-45fb-a01b-783e9a798cdc\") "
Sep 09 16:48:53.279707 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279626    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3520d030-07ef-470a-b624-7f52dd765be3" (UID: "3520d030-07ef-470a-b624-7f52dd765be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:53.279772 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279695    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba89743d-3a8d-45fb-a01b-783e9a798cdc" (UID: "ba89743d-3a8d-45fb-a01b-783e9a798cdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:48:53.279772 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.279730    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3520d030-07ef-470a-b624-7f52dd765be3-plugins-config-volume\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.281471 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.281437    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj" (OuterVolumeSpecName: "kube-api-access-492qj") pod "3520d030-07ef-470a-b624-7f52dd765be3" (UID: "3520d030-07ef-470a-b624-7f52dd765be3"). InnerVolumeSpecName "kube-api-access-492qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:53.281471 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.281449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m" (OuterVolumeSpecName: "kube-api-access-mks4m") pod "ba89743d-3a8d-45fb-a01b-783e9a798cdc" (UID: "ba89743d-3a8d-45fb-a01b-783e9a798cdc"). InnerVolumeSpecName "kube-api-access-mks4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:48:53.281629 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.281531    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ba89743d-3a8d-45fb-a01b-783e9a798cdc" (UID: "ba89743d-3a8d-45fb-a01b-783e9a798cdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:48:53.281629 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.281543    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba89743d-3a8d-45fb-a01b-783e9a798cdc" (UID: "ba89743d-3a8d-45fb-a01b-783e9a798cdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:48:53.380523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.380476    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba89743d-3a8d-45fb-a01b-783e9a798cdc-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.380523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.380495    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mks4m\" (UniqueName: \"kubernetes.io/projected/ba89743d-3a8d-45fb-a01b-783e9a798cdc-kube-api-access-mks4m\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.380523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.380505    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492qj\" (UniqueName: \"kubernetes.io/projected/3520d030-07ef-470a-b624-7f52dd765be3-kube-api-access-492qj\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.380523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.380516    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.380523 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.380523    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba89743d-3a8d-45fb-a01b-783e9a798cdc-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:48:53.849578 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.849548    2559 generic.go:356] "Generic (PLEG): container finished" podID="3520d030-07ef-470a-b624-7f52dd765be3" containerID="f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6" exitCode=0
Sep 09 16:48:53.849954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.849581    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"
Sep 09 16:48:53.849954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.849627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" event={"ID":"3520d030-07ef-470a-b624-7f52dd765be3","Type":"ContainerDied","Data":"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"}
Sep 09 16:48:53.849954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.849681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d" event={"ID":"3520d030-07ef-470a-b624-7f52dd765be3","Type":"ContainerDied","Data":"db09f257833e2a62d5601351bf0fe2087f3daa8565b0f2af50b79052503eee0f"}
Sep 09 16:48:53.849954 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.849709    2559 scope.go:117] "RemoveContainer" containerID="f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"
Sep 09 16:48:53.851302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.850987    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" containerID="1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305" exitCode=0
Sep 09 16:48:53.851302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.851025    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"
Sep 09 16:48:53.851302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.851063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" event={"ID":"ba89743d-3a8d-45fb-a01b-783e9a798cdc","Type":"ContainerDied","Data":"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"}
Sep 09 16:48:53.851302 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.851119    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb" event={"ID":"ba89743d-3a8d-45fb-a01b-783e9a798cdc","Type":"ContainerDied","Data":"088999a7d0cc42f39bfc2ea259dac47565ba727f065ecd5753ab6c4127347b0e"}
Sep 09 16:48:53.866206 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.866194    2559 scope.go:117] "RemoveContainer" containerID="f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"
Sep 09 16:48:53.866451 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:48:53.866433    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6\": container with ID starting with f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6 not found: ID does not exist" containerID="f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"
Sep 09 16:48:53.866487 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.866460    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6"} err="failed to get container status \"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6\": rpc error: code = NotFound desc = could not find container \"f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6\": container with ID starting with f99ab4a44f790fd2456b1755374ef7b12bd0121e2557d9de1212ee0c5641a8f6 not found: ID does not exist"
Sep 09 16:48:53.866487 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.866475    2559 scope.go:117] "RemoveContainer" containerID="1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"
Sep 09 16:48:53.873716 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.873703    2559 scope.go:117] "RemoveContainer" containerID="1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"
Sep 09 16:48:53.873951 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:48:53.873935    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305\": container with ID starting with 1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305 not found: ID does not exist" containerID="1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"
Sep 09 16:48:53.874022 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.873970    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305"} err="failed to get container status \"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305\": rpc error: code = NotFound desc = could not find container \"1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305\": container with ID starting with 1aef98dfa13ad343cb285682bca423fb44198dd6397c5ca0ffbe449cfc7e0305 not found: ID does not exist"
Sep 09 16:48:53.876669 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.876649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:48:53.881059 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.881041    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-331bf8-5cb4c487d4-ljplb"]
Sep 09 16:48:53.889755 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.889738    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:48:53.893044 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:53.893027    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-331bf8-579b6cf746-pfv4d"]
Sep 09 16:48:54.531172 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:54.531141    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3520d030-07ef-470a-b624-7f52dd765be3" path="/var/lib/kubelet/pods/3520d030-07ef-470a-b624-7f52dd765be3/volumes"
Sep 09 16:48:54.531516 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:48:54.531503    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" path="/var/lib/kubelet/pods/ba89743d-3a8d-45fb-a01b-783e9a798cdc/volumes"
Sep 09 16:49:10.075383 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.075351    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:49:10.075974 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.075543    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" podUID="75310ee7-5020-4aed-98d1-e2b0f9690826" containerName="maas-api" containerID="cri-o://318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524" gracePeriod=30
Sep 09 16:49:10.306343 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.306315    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:49:10.405856 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.405800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm5tl\" (UniqueName: \"kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl\") pod \"75310ee7-5020-4aed-98d1-e2b0f9690826\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") "
Sep 09 16:49:10.405856 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.405837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca\") pod \"75310ee7-5020-4aed-98d1-e2b0f9690826\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") "
Sep 09 16:49:10.406023 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.405865    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") pod \"75310ee7-5020-4aed-98d1-e2b0f9690826\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") "
Sep 09 16:49:10.406023 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.405995    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs\") pod \"75310ee7-5020-4aed-98d1-e2b0f9690826\" (UID: \"75310ee7-5020-4aed-98d1-e2b0f9690826\") "
Sep 09 16:49:10.406238 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.406210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75310ee7-5020-4aed-98d1-e2b0f9690826" (UID: "75310ee7-5020-4aed-98d1-e2b0f9690826"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:49:10.407850 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.407812    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75310ee7-5020-4aed-98d1-e2b0f9690826" (UID: "75310ee7-5020-4aed-98d1-e2b0f9690826"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:49:10.407934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.407896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "75310ee7-5020-4aed-98d1-e2b0f9690826" (UID: "75310ee7-5020-4aed-98d1-e2b0f9690826"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:49:10.407934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.407908    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl" (OuterVolumeSpecName: "kube-api-access-rm5tl") pod "75310ee7-5020-4aed-98d1-e2b0f9690826" (UID: "75310ee7-5020-4aed-98d1-e2b0f9690826"). InnerVolumeSpecName "kube-api-access-rm5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:49:10.507285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.507255    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.507285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.507283    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5tl\" (UniqueName: \"kubernetes.io/projected/75310ee7-5020-4aed-98d1-e2b0f9690826-kube-api-access-rm5tl\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.507285 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.507293    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310ee7-5020-4aed-98d1-e2b0f9690826-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.507452 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.507301    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75310ee7-5020-4aed-98d1-e2b0f9690826-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:49:10.930018 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.929988    2559 generic.go:356] "Generic (PLEG): container finished" podID="75310ee7-5020-4aed-98d1-e2b0f9690826" containerID="318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524" exitCode=0
Sep 09 16:49:10.930125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.930026    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"
Sep 09 16:49:10.930125 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.930025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" event={"ID":"75310ee7-5020-4aed-98d1-e2b0f9690826","Type":"ContainerDied","Data":"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"}
Sep 09 16:49:10.930202 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.930129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t" event={"ID":"75310ee7-5020-4aed-98d1-e2b0f9690826","Type":"ContainerDied","Data":"b3353de3e623ec7721288d8624dcc91887cd9e5685f98796935765ada91f2356"}
Sep 09 16:49:10.930202 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.930161    2559 scope.go:117] "RemoveContainer" containerID="318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"
Sep 09 16:49:10.938741 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.938726    2559 scope.go:117] "RemoveContainer" containerID="318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"
Sep 09 16:49:10.939011 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:49:10.938993    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524\": container with ID starting with 318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524 not found: ID does not exist" containerID="318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"
Sep 09 16:49:10.939089 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.939017    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524"} err="failed to get container status \"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524\": rpc error: code = NotFound desc = could not find container \"318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524\": container with ID starting with 318fb65956a336fbb237a159f6aea3fe1217e634c5d32d187bc5abe7aa3e1524 not found: ID does not exist"
Sep 09 16:49:10.950706 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.950685    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:49:10.953947 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:10.953927    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-50fcbc-79d448bffc-znf7t"]
Sep 09 16:49:12.530447 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:49:12.530414    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75310ee7-5020-4aed-98d1-e2b0f9690826" path="/var/lib/kubelet/pods/75310ee7-5020-4aed-98d1-e2b0f9690826/volumes"
Sep 09 16:53:41.552244 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:53:41.552216    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:53:41.558230 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:53:41.558210    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:54:26.006037 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006009    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006311    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006322    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006334    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f7c24f-c704-4412-bcff-85b560db9125" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006339    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f7c24f-c704-4412-bcff-85b560db9125" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006348    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006363    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3520d030-07ef-470a-b624-7f52dd765be3" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006368    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3520d030-07ef-470a-b624-7f52dd765be3" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" containerName="main"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006379    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" containerName="main"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006393    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75310ee7-5020-4aed-98d1-e2b0f9690826" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006398    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="75310ee7-5020-4aed-98d1-e2b0f9690826" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006405    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006410    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" containerName="payload-pre-processing"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006457    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="938b19e7-71ce-4dd9-85fa-61bb50f2df71" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006465    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba89743d-3a8d-45fb-a01b-783e9a798cdc" containerName="maas-api"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006470    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ffefb0-57fb-45de-9607-32a9b2df33c6" containerName="main"
Sep 09 16:54:26.006469 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006477    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c1a7ec-75ee-4c1a-bc98-c1ec5c11ab14" containerName="payload-pre-processing"
Sep 09 16:54:26.007039 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006485    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3520d030-07ef-470a-b624-7f52dd765be3" containerName="payload-pre-processing"
Sep 09 16:54:26.007039 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006491    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="75310ee7-5020-4aed-98d1-e2b0f9690826" containerName="maas-api"
Sep 09 16:54:26.007039 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.006495    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f7c24f-c704-4412-bcff-85b560db9125" containerName="payload-pre-processing"
Sep 09 16:54:26.009412 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.009392    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.013075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.013052    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 16:54:26.013075 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.013070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k96v8\""
Sep 09 16:54:26.013245 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.013116    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 16:54:26.013296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.013265    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 16:54:26.013352 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.013303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 16:54:26.018679 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.018654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:26.160021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.159996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.160147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.160038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.160147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.160072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.160147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.160124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9z2m\" (UniqueName: \"kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.261358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.261299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.261358 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.261342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.261538 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.261373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.261538 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.261422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9z2m\" (UniqueName: \"kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.262030 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.262005    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.263719 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.263701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.263794 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.263743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.268731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.268709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9z2m\" (UniqueName: \"kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m\") pod \"maas-api-7f975847c4-kx2kk\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.320146 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.320124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:26.588811 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.588791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:26.593807 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:54:26.593780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5b6685_f996_485e_9746_b81aaa7dd819.slice/crio-0da9e7a193679f90b56449e785363b165bcce1daf22b11eb8b03bd17c4fce63e WatchSource:0}: Error finding container 0da9e7a193679f90b56449e785363b165bcce1daf22b11eb8b03bd17c4fce63e: Status 404 returned error can't find the container with id 0da9e7a193679f90b56449e785363b165bcce1daf22b11eb8b03bd17c4fce63e
Sep 09 16:54:26.595148 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:26.595131    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 16:54:27.299784 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:27.299748    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" event={"ID":"cb5b6685-f996-485e-9746-b81aaa7dd819","Type":"ContainerStarted","Data":"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"}
Sep 09 16:54:27.300183 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:27.299806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" event={"ID":"cb5b6685-f996-485e-9746-b81aaa7dd819","Type":"ContainerStarted","Data":"0da9e7a193679f90b56449e785363b165bcce1daf22b11eb8b03bd17c4fce63e"}
Sep 09 16:54:27.300183 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:27.299917    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:27.315508 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:27.315423    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" podStartSLOduration=1.852289879 podStartE2EDuration="2.315408984s" totalImagesPullingTime="463.119105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:54:27.315048805 +0000 UTC m=+2447.264629650" watchObservedRunningTime="2026-09-09 16:54:27.315408984 +0000 UTC m=+2447.264989830"
Sep 09 16:54:33.312350 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:33.312316    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:34.031653 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.031623    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"]
Sep 09 16:54:34.035057 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.035042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.044896 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.044865    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"]
Sep 09 16:54:34.214731 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.214701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.214867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.214743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-metrics-certs\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.214867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.214766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ca169a-c64d-40e9-8623-38ac4e21271c-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.214867 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.214839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fg4\" (UniqueName: \"kubernetes.io/projected/95ca169a-c64d-40e9-8623-38ac4e21271c-kube-api-access-47fg4\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.316071 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.316011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.316071 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.316050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-metrics-certs\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.316071 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.316070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ca169a-c64d-40e9-8623-38ac4e21271c-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.316446 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.316186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fg4\" (UniqueName: \"kubernetes.io/projected/95ca169a-c64d-40e9-8623-38ac4e21271c-kube-api-access-47fg4\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.316714 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.316696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95ca169a-c64d-40e9-8623-38ac4e21271c-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.318382 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.318353    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.318544 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.318523    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95ca169a-c64d-40e9-8623-38ac4e21271c-metrics-certs\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.323904 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.323882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fg4\" (UniqueName: \"kubernetes.io/projected/95ca169a-c64d-40e9-8623-38ac4e21271c-kube-api-access-47fg4\") pod \"maas-api-6d6c5d7bdb-fdvz4\" (UID: \"95ca169a-c64d-40e9-8623-38ac4e21271c\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.346941 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.346926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:34.612349 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:34.612283    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"]
Sep 09 16:54:34.615418 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:54:34.615393    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ca169a_c64d_40e9_8623_38ac4e21271c.slice/crio-4d531a48c4270f9f30b12dbf60b977518ab408be640a7e0596a2e5f95d6439ca WatchSource:0}: Error finding container 4d531a48c4270f9f30b12dbf60b977518ab408be640a7e0596a2e5f95d6439ca: Status 404 returned error can't find the container with id 4d531a48c4270f9f30b12dbf60b977518ab408be640a7e0596a2e5f95d6439ca
Sep 09 16:54:35.338699 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:35.338669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4" event={"ID":"95ca169a-c64d-40e9-8623-38ac4e21271c","Type":"ContainerStarted","Data":"4d531a48c4270f9f30b12dbf60b977518ab408be640a7e0596a2e5f95d6439ca"}
Sep 09 16:54:37.348813 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:37.348775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4" event={"ID":"95ca169a-c64d-40e9-8623-38ac4e21271c","Type":"ContainerStarted","Data":"deeca616641143b4a170d01ccb3818e20f8b2a32095da5885d5e0f6f3dc05723"}
Sep 09 16:54:37.349269 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:37.348905    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:37.364021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:37.363953    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4" podStartSLOduration=1.340474285 podStartE2EDuration="3.363938351s" totalImagesPullingTime="2.023464066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 16:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:54:37.363488954 +0000 UTC m=+2457.313069844" watchObservedRunningTime="2026-09-09 16:54:37.363938351 +0000 UTC m=+2457.313519198"
Sep 09 16:54:43.358147 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:43.358116    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-fdvz4"
Sep 09 16:54:43.403450 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:43.403422    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:43.403619 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:43.403582    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" podUID="cb5b6685-f996-485e-9746-b81aaa7dd819" containerName="maas-api" containerID="cri-o://654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d" gracePeriod=30
Sep 09 16:54:44.641908 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.641877    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:44.695760 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.695729    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9z2m\" (UniqueName: \"kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m\") pod \"cb5b6685-f996-485e-9746-b81aaa7dd819\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") "
Sep 09 16:54:44.695905 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.695769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs\") pod \"cb5b6685-f996-485e-9746-b81aaa7dd819\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") "
Sep 09 16:54:44.695905 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.695788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca\") pod \"cb5b6685-f996-485e-9746-b81aaa7dd819\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") "
Sep 09 16:54:44.695905 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.695852    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls\") pod \"cb5b6685-f996-485e-9746-b81aaa7dd819\" (UID: \"cb5b6685-f996-485e-9746-b81aaa7dd819\") "
Sep 09 16:54:44.696224 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.696195    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb5b6685-f996-485e-9746-b81aaa7dd819" (UID: "cb5b6685-f996-485e-9746-b81aaa7dd819"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 16:54:44.697621 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.697596    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb5b6685-f996-485e-9746-b81aaa7dd819" (UID: "cb5b6685-f996-485e-9746-b81aaa7dd819"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:44.697733 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.697719    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m" (OuterVolumeSpecName: "kube-api-access-g9z2m") pod "cb5b6685-f996-485e-9746-b81aaa7dd819" (UID: "cb5b6685-f996-485e-9746-b81aaa7dd819"). InnerVolumeSpecName "kube-api-access-g9z2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 16:54:44.697778 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.697749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb5b6685-f996-485e-9746-b81aaa7dd819" (UID: "cb5b6685-f996-485e-9746-b81aaa7dd819"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 16:54:44.796881 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.796863    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-maas-api-tls\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:54:44.796979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.796882    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9z2m\" (UniqueName: \"kubernetes.io/projected/cb5b6685-f996-485e-9746-b81aaa7dd819-kube-api-access-g9z2m\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:54:44.796979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.796894    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb5b6685-f996-485e-9746-b81aaa7dd819-metrics-certs\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:54:44.796979 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:44.796902    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb5b6685-f996-485e-9746-b81aaa7dd819-openshift-service-ca\") on node \"ip-10-0-133-104.ec2.internal\" DevicePath \"\""
Sep 09 16:54:45.383812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.383787    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb5b6685-f996-485e-9746-b81aaa7dd819" containerID="654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d" exitCode=0
Sep 09 16:54:45.383934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.383830    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"
Sep 09 16:54:45.383934 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.383835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" event={"ID":"cb5b6685-f996-485e-9746-b81aaa7dd819","Type":"ContainerDied","Data":"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"}
Sep 09 16:54:45.384081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.383938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk" event={"ID":"cb5b6685-f996-485e-9746-b81aaa7dd819","Type":"ContainerDied","Data":"0da9e7a193679f90b56449e785363b165bcce1daf22b11eb8b03bd17c4fce63e"}
Sep 09 16:54:45.384081 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.383969    2559 scope.go:117] "RemoveContainer" containerID="654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"
Sep 09 16:54:45.392417 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.392397    2559 scope.go:117] "RemoveContainer" containerID="654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"
Sep 09 16:54:45.392659 ip-10-0-133-104 kubenswrapper[2559]: E0909 16:54:45.392640    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d\": container with ID starting with 654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d not found: ID does not exist" containerID="654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"
Sep 09 16:54:45.392725 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.392669    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d"} err="failed to get container status \"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d\": rpc error: code = NotFound desc = could not find container \"654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d\": container with ID starting with 654295defedd1343a29c655f4a5ceade20ec1e7cdcf17d63f3ab3696afdeb91d not found: ID does not exist"
Sep 09 16:54:45.404084 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.404062    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:45.407664 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:45.407644    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-kx2kk"]
Sep 09 16:54:46.529891 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:46.529859    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5b6685-f996-485e-9746-b81aaa7dd819" path="/var/lib/kubelet/pods/cb5b6685-f996-485e-9746-b81aaa7dd819/volumes"
Sep 09 16:54:55.928609 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:54:55.928578    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b544d5b74-4rnmr_b93bbb15-4526-42eb-b1ba-f789de04f939/authorino/0.log"
Sep 09 16:55:02.005439 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:02.005411    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-gsv4t_4e72ed5a-0443-4c43-bbaa-005584ec8cfc/manager/0.log"
Sep 09 16:55:02.010252 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:02.010234    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-gsv4t_4e72ed5a-0443-4c43-bbaa-005584ec8cfc/copy-manifests/0.log"
Sep 09 16:55:02.574085 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:02.574056    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d6c5d7bdb-fdvz4_95ca169a-c64d-40e9-8623-38ac4e21271c/maas-api/0.log"
Sep 09 16:55:03.739412 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:03.739386    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b544d5b74-4rnmr_b93bbb15-4526-42eb-b1ba-f789de04f939/authorino/0.log"
Sep 09 16:55:04.073474 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:04.073454    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-sc2xk_98249b20-e06f-4107-8aee-96a8fbeaa314/kuadrant-console-plugin/0.log"
Sep 09 16:55:04.450111 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:04.450049    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2wnv6_9e4c6919-ac14-41ed-a9c9-ab26e265267c/manager/0.log"
Sep 09 16:55:05.114400 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:05.114378    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-w77ms_eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50/kube-auth-proxy/0.log"
Sep 09 16:55:06.005871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:06.005841    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-m4gnz_f9c2090d-47c1-4dcb-9c73-109b191756d6/main/0.log"
Sep 09 16:55:06.221455 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:06.221433    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tblx9_cfeca21e-b5de-4dd9-bb17-fb306babebbf/main/0.log"
Sep 09 16:55:06.457556 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:06.457535    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vngp9_bfdffb26-b427-4c61-8e1c-807cb9e11c24/main/0.log"
Sep 09 16:55:14.919552 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:14.919517    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5btpt_991f1b5b-2996-488b-9cc9-45ede6430c6f/global-pull-secret-syncer/0.log"
Sep 09 16:55:15.095815 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:15.095783    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hqpjn_bf12ff35-866d-4c47-ab43-d5c2a7946d5a/konnectivity-agent/0.log"
Sep 09 16:55:15.132488 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:15.132467    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-104.ec2.internal_92a982754da1ab91809fe032c16435bd/haproxy/0.log"
Sep 09 16:55:19.738184 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:19.738151    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b544d5b74-4rnmr_b93bbb15-4526-42eb-b1ba-f789de04f939/authorino/0.log"
Sep 09 16:55:19.877871 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:19.877844    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-sc2xk_98249b20-e06f-4107-8aee-96a8fbeaa314/kuadrant-console-plugin/0.log"
Sep 09 16:55:20.070150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:20.070130    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-2wnv6_9e4c6919-ac14-41ed-a9c9-ab26e265267c/manager/0.log"
Sep 09 16:55:21.477816 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.477790    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/alertmanager/0.log"
Sep 09 16:55:21.500114 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.500091    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/config-reloader/0.log"
Sep 09 16:55:21.514810 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.514792    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/kube-rbac-proxy-web/0.log"
Sep 09 16:55:21.531812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.531793    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/kube-rbac-proxy/0.log"
Sep 09 16:55:21.549150 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.549131    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/kube-rbac-proxy-metric/0.log"
Sep 09 16:55:21.566598 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.566580    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/prom-label-proxy/0.log"
Sep 09 16:55:21.586422 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.586403    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a33105c4-d146-423f-b135-67430ac6c250/init-config-reloader/0.log"
Sep 09 16:55:21.654337 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.654317    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fcsgn_a60e7d91-7675-4ef9-b4f4-1b0aae7517e8/kube-state-metrics/0.log"
Sep 09 16:55:21.674017 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.673999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fcsgn_a60e7d91-7675-4ef9-b4f4-1b0aae7517e8/kube-rbac-proxy-main/0.log"
Sep 09 16:55:21.693296 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.693278    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fcsgn_a60e7d91-7675-4ef9-b4f4-1b0aae7517e8/kube-rbac-proxy-self/0.log"
Sep 09 16:55:21.754953 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.754939    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-nfs4j_462130c6-8e4c-4397-8d04-b317f1cbcc7a/monitoring-plugin/0.log"
Sep 09 16:55:21.817220 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.817203    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27wdt_7009d4af-e4c2-491a-8135-ae30ff707f06/node-exporter/0.log"
Sep 09 16:55:21.834808 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.834788    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27wdt_7009d4af-e4c2-491a-8135-ae30ff707f06/kube-rbac-proxy/0.log"
Sep 09 16:55:21.857897 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:21.857883    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-27wdt_7009d4af-e4c2-491a-8135-ae30ff707f06/init-textfile/0.log"
Sep 09 16:55:22.265391 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.265367    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rpk4m_a7155a4c-fded-4a56-9b8f-797fd4c10444/prometheus-operator/0.log"
Sep 09 16:55:22.278812 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.278787    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rpk4m_a7155a4c-fded-4a56-9b8f-797fd4c10444/kube-rbac-proxy/0.log"
Sep 09 16:55:22.421021 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.421000    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/thanos-query/0.log"
Sep 09 16:55:22.438047 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.438028    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/kube-rbac-proxy-web/0.log"
Sep 09 16:55:22.455522 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.455495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/kube-rbac-proxy/0.log"
Sep 09 16:55:22.476047 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.476025    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/prom-label-proxy/0.log"
Sep 09 16:55:22.495578 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.495557    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/kube-rbac-proxy-rules/0.log"
Sep 09 16:55:22.512463 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:22.512443    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66875b9776-drjwh_a9b9f82f-6815-4a95-89dd-c8a666bc5f35/kube-rbac-proxy-metrics/0.log"
Sep 09 16:55:24.825503 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.825472    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"]
Sep 09 16:55:24.825837 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.825802    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5b6685-f996-485e-9746-b81aaa7dd819" containerName="maas-api"
Sep 09 16:55:24.825837 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.825815    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5b6685-f996-485e-9746-b81aaa7dd819" containerName="maas-api"
Sep 09 16:55:24.825906 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.825877    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5b6685-f996-485e-9746-b81aaa7dd819" containerName="maas-api"
Sep 09 16:55:24.828897 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.828877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:24.831174 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.831154    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8s7bt\"/\"openshift-service-ca.crt\""
Sep 09 16:55:24.832399 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.832380    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8s7bt\"/\"kube-root-ca.crt\""
Sep 09 16:55:24.832493 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.832382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8s7bt\"/\"default-dockercfg-jl267\""
Sep 09 16:55:24.835405 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.835172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"]
Sep 09 16:55:24.986670 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.986647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-podres\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:24.986765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.986693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-lib-modules\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:24.986765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.986739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-sys\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:24.986765 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.986752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-proc\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:24.986868 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:24.986766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpqv\" (UniqueName: \"kubernetes.io/projected/6c1aada5-f7b6-4a02-9837-123456e2ef74-kube-api-access-zfpqv\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087209 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-sys\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087209 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-proc\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087209 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpqv\" (UniqueName: \"kubernetes.io/projected/6c1aada5-f7b6-4a02-9837-123456e2ef74-kube-api-access-zfpqv\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087251    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-podres\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-sys\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-proc\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-lib-modules\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087397 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-podres\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.087599 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.087464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c1aada5-f7b6-4a02-9837-123456e2ef74-lib-modules\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.094228 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.094199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpqv\" (UniqueName: \"kubernetes.io/projected/6c1aada5-f7b6-4a02-9837-123456e2ef74-kube-api-access-zfpqv\") pod \"perf-node-gather-daemonset-npgqz\" (UID: \"6c1aada5-f7b6-4a02-9837-123456e2ef74\") " pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.139200 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.139179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.406684 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.406498    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"]
Sep 09 16:55:25.409231 ip-10-0-133-104 kubenswrapper[2559]: W0909 16:55:25.409206    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6c1aada5_f7b6_4a02_9837_123456e2ef74.slice/crio-15bff526f45a54aea9661a3b1277918f3544807daa9931f116e1389b88968486 WatchSource:0}: Error finding container 15bff526f45a54aea9661a3b1277918f3544807daa9931f116e1389b88968486: Status 404 returned error can't find the container with id 15bff526f45a54aea9661a3b1277918f3544807daa9931f116e1389b88968486
Sep 09 16:55:25.559403 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.559376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz" event={"ID":"6c1aada5-f7b6-4a02-9837-123456e2ef74","Type":"ContainerStarted","Data":"f588f8df5239a0b03e3c5e48419d2fca8054b00df81ebb494a8e152fffd8cd51"}
Sep 09 16:55:25.559495 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.559421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz" event={"ID":"6c1aada5-f7b6-4a02-9837-123456e2ef74","Type":"ContainerStarted","Data":"15bff526f45a54aea9661a3b1277918f3544807daa9931f116e1389b88968486"}
Sep 09 16:55:25.559535 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.559508    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:25.575639 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:25.573779    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz" podStartSLOduration=1.573763765 podStartE2EDuration="1.573763765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 16:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 16:55:25.573325959 +0000 UTC m=+2505.522906804" watchObservedRunningTime="2026-09-09 16:55:25.573763765 +0000 UTC m=+2505.523344613"
Sep 09 16:55:26.629762 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:26.629736    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x9bq_cb2d6b57-09e3-438f-be2a-93fcc6aea9e5/dns/0.log"
Sep 09 16:55:26.646901 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:26.646880    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5x9bq_cb2d6b57-09e3-438f-be2a-93fcc6aea9e5/kube-rbac-proxy/0.log"
Sep 09 16:55:26.786515 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:26.786494    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cm49p_6d07d2f8-cd59-4b61-8b13-27675f985d5b/dns-node-resolver/0.log"
Sep 09 16:55:27.301131 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:27.301103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9w9h5_2821554a-7a86-463f-a3bf-78c92fd9cba0/node-ca/0.log"
Sep 09 16:55:28.549141 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:28.549117    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-w77ms_eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50/kube-auth-proxy/0.log"
Sep 09 16:55:29.468651 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:29.468602    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhfmc_9f61d873-8851-48cb-9b3d-5243316b65a8/serve-healthcheck-canary/0.log"
Sep 09 16:55:30.003804 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:30.003779    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-94ss4_44341d7e-e18f-416c-9eee-ecaa4010342b/kube-rbac-proxy/0.log"
Sep 09 16:55:30.030891 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:30.030870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-94ss4_44341d7e-e18f-416c-9eee-ecaa4010342b/exporter/0.log"
Sep 09 16:55:30.048634 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:30.048614    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-94ss4_44341d7e-e18f-416c-9eee-ecaa4010342b/extractor/0.log"
Sep 09 16:55:31.572293 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:31.572268    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8s7bt/perf-node-gather-daemonset-npgqz"
Sep 09 16:55:32.554908 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:32.554877    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-gsv4t_4e72ed5a-0443-4c43-bbaa-005584ec8cfc/manager/0.log"
Sep 09 16:55:32.569761 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:32.569744    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b9b77fc68-gsv4t_4e72ed5a-0443-4c43-bbaa-005584ec8cfc/copy-manifests/0.log"
Sep 09 16:55:33.673732 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:33.673668    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6bb9f9f6c8-fsh5j_aab139f6-c5b9-443f-83a0-9a8760b10173/manager/0.log"
Sep 09 16:55:34.817668 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:34.817644    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-w77ms_eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50/kube-auth-proxy/0.log"
Sep 09 16:55:36.957569 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:36.957496    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769d969b5b-w77ms_eaa0f8c7-dc07-4d54-8df5-b1ea2e06eb50/kube-auth-proxy/0.log"
Sep 09 16:55:40.200870 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.200839    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/kube-multus-additional-cni-plugins/0.log"
Sep 09 16:55:40.217010 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.216987    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/egress-router-binary-copy/0.log"
Sep 09 16:55:40.233222 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.233201    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/cni-plugins/0.log"
Sep 09 16:55:40.256038 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.256017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/bond-cni-plugin/0.log"
Sep 09 16:55:40.272240 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.272226    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/routeoverride-cni/0.log"
Sep 09 16:55:40.288675 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.288659    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/whereabouts-cni-bincopy/0.log"
Sep 09 16:55:40.305507 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.305482    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mqmsm_cd6a48e2-28d7-4fa6-b143-75a8f7c11858/whereabouts-cni/0.log"
Sep 09 16:55:40.351373 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.351350    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cnfqg_9c0a1341-9866-4678-acf3-753b34a41424/kube-multus/0.log"
Sep 09 16:55:40.441385 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.441367    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mgf65_c6337926-86fe-44e6-82e4-2e7637dc1b70/network-metrics-daemon/0.log"
Sep 09 16:55:40.456092 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:40.456045    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mgf65_c6337926-86fe-44e6-82e4-2e7637dc1b70/kube-rbac-proxy/0.log"
Sep 09 16:55:41.414320 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.414293    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-controller/0.log"
Sep 09 16:55:41.427720 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.427696    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/1.log"
Sep 09 16:55:41.427848 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.427730    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovn-acl-logging/0.log"
Sep 09 16:55:41.450557 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.450534    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/kube-rbac-proxy-node/0.log"
Sep 09 16:55:41.472024 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.472004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 16:55:41.485108 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.485088    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/northd/0.log"
Sep 09 16:55:41.500903 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.500886    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/nbdb/0.log"
Sep 09 16:55:41.520135 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.520079    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/sbdb/0.log"
Sep 09 16:55:41.598662 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:41.598637    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bxkn8_60331c31-f2d3-4dff-9ff9-b204a72ff1fc/ovnkube-controller/0.log"
Sep 09 16:55:42.790926 ip-10-0-133-104 kubenswrapper[2559]: I0909 16:55:42.790898    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-f5tf2_f27803b5-592e-4448-94d5-7009bdb04824/network-check-target-container/0.log"