Sep 10 11:51:37.101940 ip-10-0-134-58 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 11:51:37.526422 ip-10-0-134-58 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:51:37.526422 ip-10-0-134-58 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 11:51:37.526422 ip-10-0-134-58 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:51:37.526422 ip-10-0-134-58 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 11:51:37.526422 ip-10-0-134-58 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 11:51:37.527110 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.526489    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 11:51:37.530890 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530872    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.530890 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530889    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530893    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530896    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530899    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530902    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530905    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530908    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530913    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530915    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530918    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530921    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530924    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530927    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530930    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530933    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530936    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530940    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530944    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530947    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530950    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.530957 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530953    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530955    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530958    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530961    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530964    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530967    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530970    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530973    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530976    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530978    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530981    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530983    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530986    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530988    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530991    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530994    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530996    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.530999    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531002    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531005    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.531425 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531007    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531010    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531012    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531015    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531017    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531020    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531022    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531025    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531027    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531030    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531032    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531035    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531037    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531040    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531044    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531047    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531049    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531052    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531055    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531058    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.531974 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531060    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531063    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531065    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531068    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531071    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531073    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531076    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531079    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531081    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531084    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531086    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531089    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531092    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531094    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531097    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531101    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531106    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531110    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531112    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.532451 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531115    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531118    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531122    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531126    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531129    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531132    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531134    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531138    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531141    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531144    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531148    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531151    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531154    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531157    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531159    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531162    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531165    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531168    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531170    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.532920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531900    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531907    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531910    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531913    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531916    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531919    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531926    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531929    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531932    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531934    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531937    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531940    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531943    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531945    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531948    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531951    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531954    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531957    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531959    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531964    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.533375 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531969    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531973    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531976    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531979    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531983    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531986    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531989    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531992    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531994    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.531997    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532000    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532002    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532005    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532007    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532010    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532012    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532015    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532017    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532020    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532023    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.533881 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532026    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532029    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532031    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532034    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532036    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532039    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532041    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532044    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532046    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532049    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532052    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532054    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532058    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532060    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532063    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532065    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532068    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532071    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532074    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.534397 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532076    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532079    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532081    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532084    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532087    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532089    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532092    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532094    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532097    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532099    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532102    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532104    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532107    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532110    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532113    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532115    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532118    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532120    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532123    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532127    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.534875 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532131    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532134    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532136    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532139    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532142    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532145    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532148    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532150    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532153    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532157    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532161    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532164    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532167    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532170    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532173    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532176    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532178    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532181    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532184    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.535355 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532186    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532569    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532574    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532577    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532580    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532583    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532587    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532589    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532592    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532595    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532597    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532601    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532605    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532607    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532610    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532613    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532632    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532635    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532638    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.535835 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532641    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532644    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532646    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532649    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532652    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532654    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532657    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532660    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532662    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532665    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532667    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532670    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532673    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532675    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532678    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532681    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532684    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532686    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532689    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.536299 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532692    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532695    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532697    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532700    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532702    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532705    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532708    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532710    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532713    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532715    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532720    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532723    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532726    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532729    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532732    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532735    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532737    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532740    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532743    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.536869 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532746    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532748    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532750    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532753    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532756    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532758    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532761    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532763    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532765    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532768    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532771    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532773    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532775    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532778    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532780    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532784    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532786    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532789    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532791    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532793    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.537336 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532798    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532801    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532804    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532807    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532810    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532813    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532816    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532819    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532821    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532824    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532826    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532829    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532831    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532834    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532836    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532839    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532841    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532844    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532847    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532849    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.537841 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532852    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532854    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.532857    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532919    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532954    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532959    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532964    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532968    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532975    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532979    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532983    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532987    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532990    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532994    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.532997    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533000    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533003    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533008    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533011    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533014    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533017    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533021    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533024    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533027    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 11:51:37.538340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533030    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533033    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533037    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533040    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533043    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533046    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533049    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533052    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533055    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533058    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533061    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533065    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533068    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533070    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533074    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533077    2573 flags.go:64] FLAG: --enable-server="true"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533079    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533084    2573 flags.go:64] FLAG: --event-burst="100"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533087    2573 flags.go:64] FLAG: --event-qps="50"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533090    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533093    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533096    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533099    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533102    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533105    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 11:51:37.538922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533110    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533113    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533116    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533119    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533122    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533125    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533128    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533131    2573 flags.go:64] FLAG: --feature-gates=""
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533135    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533138    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533141    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533144    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533147    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533150    2573 flags.go:64] FLAG: --help="false"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533152    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533155    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533159    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533162    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533165    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533169    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533172    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533174    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533177    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 11:51:37.539502 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533180    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533183    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533187    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533190    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533193    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533196    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533199    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533203    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533206    2573 flags.go:64] FLAG: --lock-file=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533211    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533213    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533216    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533221    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533224    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533227    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533230    2573 flags.go:64] FLAG: --logging-format="text"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533233    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533236    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533239    2573 flags.go:64] FLAG: --manifest-url=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533242    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533246    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533250    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533254    2573 flags.go:64] FLAG: --max-pods="110"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533257    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533260    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 11:51:37.540141 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533263    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533266    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533269    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533272    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533275    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533282    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533285    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533288    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533291    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533294    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533300    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533303    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533306    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533309    2573 flags.go:64] FLAG: --port="10250"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533313    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533316    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038f86f2d48892f36"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533320    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533324    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533327    2573 flags.go:64] FLAG: --register-node="true"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533330    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533334    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533337    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533340    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533342    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533346    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 11:51:37.540776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533349    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533352    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533355    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533358    2573 flags.go:64] FLAG: --runonce="false"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533361    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533363    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533367    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533369    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533372    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533375    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533378    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533381    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533384    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533387    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533390    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533392    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533395    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533398    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533401    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533407    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533409    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533413    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533417    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533420    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533424    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 11:51:37.541384 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533427    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533430    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533433    2573 flags.go:64] FLAG: --v="2"
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533437    2573 flags.go:64] FLAG: --version="false"
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533441    2573 flags.go:64] FLAG: --vmodule=""
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533444    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.533447    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533562    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533566    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533570    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533573    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533576    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533581    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533584    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533588    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533590    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533593    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533596    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533598    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533601    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533604    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.542311 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533606    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533609    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533611    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533628    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533631    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533634    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533637    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533641    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533643    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533647    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533652    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533655    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533658    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533661    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533664    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533667    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533669    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533672    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533675    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533677    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.543004 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533680    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533682    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533685    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533688    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533690    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533693    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533696    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533698    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533701    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533703    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533706    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533709    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533711    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533714    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533716    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533719    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533722    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533724    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533727    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533731    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.543501 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533734    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533737    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533740    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533743    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533746    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533748    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533751    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533753    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533756    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533758    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533761    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533763    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533768    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533771    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533774    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533777    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533780    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533782    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533785    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.544046 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533788    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533790    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533793    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533795    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533798    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533800    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533803    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533805    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533808    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533811    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533813    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533816    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533820    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533823    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533825    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533829    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533832    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533835    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533837    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533840    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.544520 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533842    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533845    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533847    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533850    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533853    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.533855    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.534645    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.544092    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 11:51:37.545014 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.544108    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 11:51:37.545834 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.545823    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 11:51:37.545867 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.545835    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 11:51:37.545909 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545901    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.545909 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545909    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545913    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545918    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545922    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545925    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545929    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545932    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545935    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545937    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545940    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545943    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545945    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545948    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545951    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545953    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545956    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545959    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545961    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545964    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.545960 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545967    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545970    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545973    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545975    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545978    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545981    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545983    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545986    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545993    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545996    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.545999    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546002    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546004    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546007    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546009    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546012    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546014    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546017    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546020    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546022    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.546432 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546025    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546027    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546030    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546032    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546034    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546037    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546040    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546043    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546045    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546048    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546050    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546053    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546056    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546058    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546062    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546065    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546076    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546080    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546084    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.546920 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546088    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546091    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546094    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546096    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546099    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546102    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546105    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546109    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546112    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546115    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546117    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546120    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546123    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546125    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546127    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546130    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546133    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546135    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546138    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546141    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.547382 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546149    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546152    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546155    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546158    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546160    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546163    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546166    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546169    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546171    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546174    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546176    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546179    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546181    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546184    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546187    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546189    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546192    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546194    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546197    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.547967 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546200    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.546204    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546329    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546334    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546337    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546339    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546342    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546344    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546347    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546350    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546352    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546355    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546357    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546360    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546368    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 11:51:37.548428 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546371    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546373    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546376    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546378    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546382    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546386    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546389    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546392    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546395    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546398    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546401    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546404    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546407    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546410    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546413    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546416    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546419    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546421    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546424    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546426    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 11:51:37.548800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546429    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546432    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546435    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546437    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546440    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546443    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546445    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546448    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546450    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546453    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546456    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546458    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546466    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546468    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546471    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546473    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546476    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546478    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546481    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546484    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 11:51:37.549278 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546487    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546489    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546492    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546494    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546497    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546500    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546502    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546505    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546507    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546510    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546512    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546515    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546518    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546520    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546522    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546525    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546527    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546530    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546534    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 11:51:37.549774 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546537    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546540    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546543    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546546    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546548    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546551    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546559    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546562    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546565    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546568    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546571    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546575    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546578    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546581    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546583    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546586    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546589    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546591    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546594    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546597    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 11:51:37.550247 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546599    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546603    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546606    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546609    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546611    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546614    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:37.546633    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.546638    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 11:51:37.550723 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.546745    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 11:51:37.553280 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.553265    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 11:51:37.554286 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.554275    2573 server.go:1019] "Starting client certificate rotation"
Sep 10 11:51:37.554393 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.554375    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 11:51:37.554425 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.554417    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 11:51:37.557094 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.557081    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 11:51:37.562557 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.562543    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 11:51:37.578509 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.578480    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 10 11:51:37.582003 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.581981    2573 log.go:25] "Validated CRI v1 image API"
Sep 10 11:51:37.582964 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.582944    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 11:51:37.585965 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.585945    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 11:51:37.588162 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.588144    2573 fs.go:135] Filesystem UUIDs: map[0b52e79b-fd2a-4a5e-a352-cb07533de4a6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 edc02c0a-6ac8-4d58-a420-45c3a77e1e41:/dev/nvme0n1p3]
Sep 10 11:51:37.588219 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.588162    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 11:51:37.595815 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.595699    2573 manager.go:216] Machine: {Timestamp:2026-09-10 11:51:37.594348146 +0000 UTC m=+0.380958106 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3158397 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29b7bf361c0706f57b108a9552c5f2 SystemUUID:ec29b7bf-361c-0706-f57b-108a9552c5f2 BootID:e45a3429-7121-406f-9ece-b985754a6750 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:50:19:37:b4:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:50:19:37:b4:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:82:82:7b:9d:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 11:51:37.595815 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.595804    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 11:51:37.595947 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.595913    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 11:51:37.599495 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.599467    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 11:51:37.599676 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.599499    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 11:51:37.600361 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.600351    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 11:51:37.600396 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.600364    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 11:51:37.600396 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.600372    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 11:51:37.601184 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.601174    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 11:51:37.602277 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.602267    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 11:51:37.606589 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606578    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 11:51:37.606730 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606721    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 11:51:37.606761 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606754    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 11:51:37.606798 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606766    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 11:51:37.606798 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606776    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 11:51:37.606798 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606786    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 10 11:51:37.606798 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.606795    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 11:51:37.607179 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.607167    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 11:51:37.607242 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.607185    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 11:51:37.610189 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.610173    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 11:51:37.616174 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.616154    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 11:51:37.616268 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.616186    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 11:51:37.616956 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.616680    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 11:51:37.616956 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.616802    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618188    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618207    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618215    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618220    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618228    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618234    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618247    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618252    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618258    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618264    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618272    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 11:51:37.618721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618282    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 11:51:37.619053 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.618769    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l76jb"
Sep 10 11:51:37.619178 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.619168    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 11:51:37.619207 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.619178    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 11:51:37.620915 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.620898    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l76jb"
Sep 10 11:51:37.623048 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.623035    2573 server.go:1268] "Started kubelet"
Sep 10 11:51:37.623183 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.623125    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 11:51:37.623234 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.623206    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 11:51:37.623284 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.623261    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 11:51:37.623937 ip-10-0-134-58 systemd[1]: Started Kubernetes Kubelet.
Sep 10 11:51:37.625235 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.625218    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 11:51:37.625997 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.625966    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 11:51:37.627680 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.627655    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 11:51:37.630395 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.630377    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 11:51:37.630395 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.630381    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 11:51:37.630544 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.630508    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:37.631176 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631157    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 11:51:37.631255 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631178    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 11:51:37.631255 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631232    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 11:51:37.631255 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631244    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 11:51:37.631401 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631305    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 11:51:37.631479 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631465    2573 factory.go:55] Registering systemd factory
Sep 10 11:51:37.631518 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631502    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 10 11:51:37.631940 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631926    2573 factory.go:153] Registering CRI-O factory
Sep 10 11:51:37.631940 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631941    2573 factory.go:223] Registration of the crio container factory successfully
Sep 10 11:51:37.632068 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631984    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 11:51:37.632068 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.631999    2573 factory.go:103] Registering Raw factory
Sep 10 11:51:37.632068 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.632009    2573 manager.go:1196] Started watching for new ooms in manager
Sep 10 11:51:37.632440 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.632430    2573 manager.go:349] Starting recovery of all containers
Sep 10 11:51:37.632796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.632775    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:37.638810 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.638762    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-58.ec2.internal\" not found" node="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.638916 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.638879    2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-58.ec2.internal" not found
Sep 10 11:51:37.644312 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.644281    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 11:51:37.644903 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.644885    2573 manager.go:354] Recovery completed
Sep 10 11:51:37.646467 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.646350    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 11:51:37.649471 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.649457    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.652829 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.652813    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.652890 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.652841    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.652890 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.652852    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.653394 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.653382    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 11:51:37.653394 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.653394    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 11:51:37.653497 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.653411    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 11:51:37.653497 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.653414    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.654167 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.654154    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.654212 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.654179    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.654212 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.654188    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.655051 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.655039    2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-58.ec2.internal" not found
Sep 10 11:51:37.655662 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.655651    2573 policy_none.go:49] "None policy: Start"
Sep 10 11:51:37.655703 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.655668    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 11:51:37.655703 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.655678    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 11:51:37.657681 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.657671    2573 policy_none.go:47] "Start"
Sep 10 11:51:37.696222 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696206    2573 manager.go:337] "Starting Device Plugin manager"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.696250    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696260    2573 server.go:87] "Starting device plugin registration server"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696521    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696535    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696651    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696733    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.696741    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.697591    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.697659    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:37.714579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.711761    2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-134-58.ec2.internal" not found
Sep 10 11:51:37.733497 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.733475    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 11:51:37.733606 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.733524    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 11:51:37.733606 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.733548    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 11:51:37.733606 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.733584    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 11:51:37.735943 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.735891    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:37.796948 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.796902    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.797983 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.797944    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.798109 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.797986    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.798109 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.798001    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.798109 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.798024    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.809275 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.809253    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.809363 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.809278    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-58.ec2.internal\": node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:37.821252 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.821230    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:37.833665 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.833634    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"]
Sep 10 11:51:37.833755 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.833700    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.835253 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.835237    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.835328 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.835263    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.835328 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.835273    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.837536 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.837524    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.837701 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.837686    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.837743 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.837717    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.838320 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838302    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.838415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838329    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.838415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838339    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.838415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838306    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.838415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838408    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.838564 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.838424    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.840541 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.840524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.840614 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.840558    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 11:51:37.841269 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.841254    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 11:51:37.841349 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.841277    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 11:51:37.841349 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.841287    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeHasSufficientPID"
Sep 10 11:51:37.859135 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.859118    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-58.ec2.internal\" not found" node="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.863683 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.863668    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-58.ec2.internal\" not found" node="ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.921698 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:37.921667    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:37.933013 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.932986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.933136 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.933018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:37.933136 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:37.933036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa9e6d39be639df8387e0cdc021c44c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-58.ec2.internal\" (UID: \"7aa9e6d39be639df8387e0cdc021c44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.022523 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.022444    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.033802 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.033870 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.033870 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa9e6d39be639df8387e0cdc021c44c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-58.ec2.internal\" (UID: \"7aa9e6d39be639df8387e0cdc021c44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.033870 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033850    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aa9e6d39be639df8387e0cdc021c44c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-58.ec2.internal\" (UID: \"7aa9e6d39be639df8387e0cdc021c44c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.033971 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.033971 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.033877    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b432bc9933fa4742789549900dceff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal\" (UID: \"f3b432bc9933fa4742789549900dceff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.123240 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.123202    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.161686 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.161650    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.167929 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.167900    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:38.223591 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.223553    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.324177 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.324095    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.424705 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.424666    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.525265 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.525235    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.553746 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.553700    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 11:51:38.554248 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.553890    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 11:51:38.554248 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.553941    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 11:51:38.626245 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.626173    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.627298 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.627257    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 11:46:38 +0000 UTC" deadline="2028-02-19 09:56:54.053435731 +0000 UTC"
Sep 10 11:51:38.627347 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.627301    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12646h5m15.426138929s"
Sep 10 11:51:38.727071 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.726901    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.809579 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:38.809536    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b432bc9933fa4742789549900dceff.slice/crio-942ee5ae86fdcfc1d28d7551874db1f10c7b19600130364d8a253ee9f91aee7a WatchSource:0}: Error finding container 942ee5ae86fdcfc1d28d7551874db1f10c7b19600130364d8a253ee9f91aee7a: Status 404 returned error can't find the container with id 942ee5ae86fdcfc1d28d7551874db1f10c7b19600130364d8a253ee9f91aee7a
Sep 10 11:51:38.809822 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:38.809805    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa9e6d39be639df8387e0cdc021c44c.slice/crio-faae484ca83c429d60a0ca9880d451b4fdf94f9e1d9d441961b8823de4b991ad WatchSource:0}: Error finding container faae484ca83c429d60a0ca9880d451b4fdf94f9e1d9d441961b8823de4b991ad: Status 404 returned error can't find the container with id faae484ca83c429d60a0ca9880d451b4fdf94f9e1d9d441961b8823de4b991ad
Sep 10 11:51:38.813597 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.813582    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:51:38.828106 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.828077    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.928759 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:38.928672    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-58.ec2.internal\" not found"
Sep 10 11:51:38.992945 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:38.992911    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:39.031336 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:39.031311    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:39.040441 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:39.040425    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 11:51:39.042281 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:39.042268    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal"
Sep 10 11:51:39.061276 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:39.061256    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 11:51:40.408931 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.408905    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:40.574993 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.574959    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:40.586815 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.586789    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 11:51:40.813903 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.813824    2573 apiserver.go:52] "Watching apiserver"
Sep 10 11:51:40.821981 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.821724    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 11:51:40.823383 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.823353    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-77jx8","openshift-network-operator/iptables-alerter-pdr7n","openshift-ovn-kubernetes/ovnkube-node-4nhr2","kube-system/global-pull-secret-syncer-j6wq6","kube-system/konnectivity-agent-vtfn5","kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw","openshift-cluster-node-tuning-operator/tuned-wtfbk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal","openshift-multus/multus-wljxn","openshift-multus/network-metrics-daemon-4brxb","openshift-dns/node-resolver-zrfl6","openshift-image-registry/node-ca-5xr7v","openshift-multus/multus-additional-cni-plugins-ktlbq"]
Sep 10 11:51:40.826265 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.826239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.829150 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.828798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.829348 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.829326    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.829348 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.829341    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-87ssd\""
Sep 10 11:51:40.829494 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.829401    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.829494 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.829328    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 11:51:40.831415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.831219    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.831512 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.831474    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 11:51:40.833326 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.831812    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.833326 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.831911    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.833326 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.831999    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q79lk\""
Sep 10 11:51:40.833813 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.833788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.833908 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:40.833847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:40.834355 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.834096    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 11:51:40.834355 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.834161    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 11:51:40.834355 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.834170    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9mhmm\""
Sep 10 11:51:40.834355 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.834186    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.834727 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.834707    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.837451 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.836841    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 11:51:40.837451 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.837210    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 11:51:40.837451 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.837370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.839474 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.838253    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 11:51:40.839474 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.839190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.840876 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.840858    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 11:51:40.840964 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.840883    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 11:51:40.841098 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.841083    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9q5nc\""
Sep 10 11:51:40.841563 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.841549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.842276 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842075    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.842276 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842094    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6jz4s\""
Sep 10 11:51:40.842276 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842265    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.843947 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842407    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 11:51:40.843947 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842484    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 11:51:40.843947 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.842611    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 11:51:40.843947 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.843795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:40.843947 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:40.843859    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:40.844581 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.844249    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p8hlq\""
Sep 10 11:51:40.844581 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.844347    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.844581 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.844555    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.847458 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.846991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:40.848038 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.848121 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-slash\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848121 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848076    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848121 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848114    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-env-overrides\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848290 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848136    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-script-lib\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848290 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02aaa613-10e8-4aca-9d8c-7b78a008d92a-agent-certs\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.848290 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02aaa613-10e8-4aca-9d8c-7b78a008d92a-konnectivity-ca\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.848290 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-etc-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848290 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-device-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-cnibin\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-socket-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxpc\" (UniqueName: \"kubernetes.io/projected/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-kube-api-access-nwxpc\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-bin\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-registration-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-kubelet\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-netns\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-config\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-dbus\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sncdb\" (UniqueName: \"kubernetes.io/projected/1c8ef79f-b500-41f7-9278-23a45dc7911e-kube-api-access-sncdb\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-serviceca\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848823    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-netd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.848850 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-systemd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-system-cni-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf84s\" (UniqueName: \"kubernetes.io/projected/28149a1e-45ad-42d4-abbf-e26de7a39e82-kube-api-access-vf84s\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjwjk\" (UniqueName: \"kubernetes.io/projected/8bf0ede9-db60-489e-a386-8c46125d0caf-kube-api-access-jjwjk\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.848999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849053    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-sys-fs\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-host\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-var-lib-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-ovn\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-node-log\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-log-socket\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-etc-selinux\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-systemd-units\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf0ede9-db60-489e-a386-8c46125d0caf-ovn-node-metrics-cert\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.849464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-kubelet-config\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.850237 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849466    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.850237 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849663    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.850237 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.849915    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j9fb5\""
Sep 10 11:51:40.850416 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.850404    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 11:51:40.851329 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.851292    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 11:51:40.851572 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.851554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zgznx\""
Sep 10 11:51:40.851659 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.851643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:40.851726 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:40.851706    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:40.851988 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.851969    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 11:51:40.853665 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.853644    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:40.858943 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.858925    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 11:51:40.859036 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.858961    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 11:51:40.859244 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.859229    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h8gnj\""
Sep 10 11:51:40.878855 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.878832    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxm57"
Sep 10 11:51:40.888222 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.888202    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxm57"
Sep 10 11:51:40.939759 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.939731    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 11:51:40.945935 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.945879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal" event={"ID":"7aa9e6d39be639df8387e0cdc021c44c","Type":"ContainerStarted","Data":"faae484ca83c429d60a0ca9880d451b4fdf94f9e1d9d441961b8823de4b991ad"}
Sep 10 11:51:40.947449 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.947417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal" event={"ID":"f3b432bc9933fa4742789549900dceff","Type":"ContainerStarted","Data":"942ee5ae86fdcfc1d28d7551874db1f10c7b19600130364d8a253ee9f91aee7a"}
Sep 10 11:51:40.949710 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949673    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.949710 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sncdb\" (UniqueName: \"kubernetes.io/projected/1c8ef79f-b500-41f7-9278-23a45dc7911e-kube-api-access-sncdb\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddfd1c58-3ab2-4b8c-b059-a720026747a2-iptables-alerter-script\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnz5\" (UniqueName: \"kubernetes.io/projected/f0632457-1698-4ba2-94de-33bf5e97f4af-kube-api-access-jcnz5\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-host\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-serviceca\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-bin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-multus\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949866    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-var-lib-kubelet\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.949892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-cnibin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-netns\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-modprobe-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-sys\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.949993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-sys-fs\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950011    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-host\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-var-lib-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-kubelet-config\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2hf\" (UniqueName: \"kubernetes.io/projected/ddfd1c58-3ab2-4b8c-b059-a720026747a2-kube-api-access-qw2hf\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-multus-certs\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-etc-selinux\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-kubelet\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0632457-1698-4ba2-94de-33bf5e97f4af-hosts-file\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:40.950236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-conf\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62fs\" (UniqueName: \"kubernetes.io/projected/fad6550a-c538-4e96-9e17-5019310b5341-kube-api-access-j62fs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-slash\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950278    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-env-overrides\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-script-lib\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-k8s-cni-cncf-io\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxpc\" (UniqueName: \"kubernetes.io/projected/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-kube-api-access-nwxpc\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-bin\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-system-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysconfig\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-netns\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-config\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-dbus\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-hostroot\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0632457-1698-4ba2-94de-33bf5e97f4af-tmp-dir\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-kubernetes\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.950796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-systemd\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-etc-tuned\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-netd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdghr\" (UniqueName: \"kubernetes.io/projected/28718450-4130-489c-a497-558b4fd200c9-kube-api-access-sdghr\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-systemd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-system-cni-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950791    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf84s\" (UniqueName: \"kubernetes.io/projected/28149a1e-45ad-42d4-abbf-e26de7a39e82-kube-api-access-vf84s\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjwjk\" (UniqueName: \"kubernetes.io/projected/8bf0ede9-db60-489e-a386-8c46125d0caf-kube-api-access-jjwjk\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-ovn\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-node-log\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-log-socket\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf0ede9-db60-489e-a386-8c46125d0caf-ovn-node-metrics-cert\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-socket-dir-parent\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.951400 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-systemd-units\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddfd1c58-3ab2-4b8c-b059-a720026747a2-host-slash\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.950980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-lib-modules\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02aaa613-10e8-4aca-9d8c-7b78a008d92a-agent-certs\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02aaa613-10e8-4aca-9d8c-7b78a008d92a-konnectivity-ca\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-cni-binary-copy\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-conf-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951119    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-run\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-tmp\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-etc-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-device-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-cnibin\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-socket-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxwb\" (UniqueName: \"kubernetes.io/projected/35b309ee-baab-4bea-9d9d-e381f60774ea-kube-api-access-wfxwb\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-registration-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.952022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-kubelet\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-daemon-config\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-etc-kubernetes\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.951612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.952711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-serviceca\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.952888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952748    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.952888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-sys-fs\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.952888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952869    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-host\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.952976 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-var-lib-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.952976 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.952976 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.952949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-kubelet-config\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.953072 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-etc-selinux\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.953131 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953110    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-slash\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.953360 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953335    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-log-socket\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.953443 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-ovn\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.953443 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-netns\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.953694 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953655    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 11:51:40.953874 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.953848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.954371 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-config\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.954470 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954449    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-socket-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.954554 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.954554 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-netd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.954700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954556    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/edc233d2-8734-410f-9add-9b30778726f8-dbus\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:40.954700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-registration-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.954700 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:40.954605    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:40.954700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954638    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-kubelet\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.954700 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:40.954666    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.454646732 +0000 UTC m=+3.034574116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:40.954952 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-run-systemd\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.954952 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.954886    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-system-cni-dir\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.955154 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.955133    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02aaa613-10e8-4aca-9d8c-7b78a008d92a-konnectivity-ca\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.955216 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.955190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-node-log\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.955270 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.955235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-systemd-units\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.955347 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.955271    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.955673 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.955651    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28149a1e-45ad-42d4-abbf-e26de7a39e82-cni-binary-copy\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956098    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-ovnkube-script-lib\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956148    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c8ef79f-b500-41f7-9278-23a45dc7911e-device-dir\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956171    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-etc-openvswitch\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf0ede9-db60-489e-a386-8c46125d0caf-host-cni-bin\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf0ede9-db60-489e-a386-8c46125d0caf-env-overrides\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.956739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.956539    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28149a1e-45ad-42d4-abbf-e26de7a39e82-cnibin\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:40.958114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.958095    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf0ede9-db60-489e-a386-8c46125d0caf-ovn-node-metrics-cert\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.958243 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.958222    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02aaa613-10e8-4aca-9d8c-7b78a008d92a-agent-certs\") pod \"konnectivity-agent-vtfn5\" (UID: \"02aaa613-10e8-4aca-9d8c-7b78a008d92a\") " pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:40.960796 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.960765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sncdb\" (UniqueName: \"kubernetes.io/projected/1c8ef79f-b500-41f7-9278-23a45dc7911e-kube-api-access-sncdb\") pod \"aws-ebs-csi-driver-node-bj7bw\" (UID: \"1c8ef79f-b500-41f7-9278-23a45dc7911e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:40.968631 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.968605    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjwjk\" (UniqueName: \"kubernetes.io/projected/8bf0ede9-db60-489e-a386-8c46125d0caf-kube-api-access-jjwjk\") pod \"ovnkube-node-4nhr2\" (UID: \"8bf0ede9-db60-489e-a386-8c46125d0caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:40.969041 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.969019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxpc\" (UniqueName: \"kubernetes.io/projected/5bee52f2-0f1c-4c43-bb6b-0520c65a8af0-kube-api-access-nwxpc\") pod \"node-ca-5xr7v\" (UID: \"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0\") " pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:40.969124 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:40.969067    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf84s\" (UniqueName: \"kubernetes.io/projected/28149a1e-45ad-42d4-abbf-e26de7a39e82-kube-api-access-vf84s\") pod \"multus-additional-cni-plugins-ktlbq\" (UID: \"28149a1e-45ad-42d4-abbf-e26de7a39e82\") " pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:41.052056 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-kubelet\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0632457-1698-4ba2-94de-33bf5e97f4af-hosts-file\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-conf\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62fs\" (UniqueName: \"kubernetes.io/projected/fad6550a-c538-4e96-9e17-5019310b5341-kube-api-access-j62fs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-k8s-cni-cncf-io\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052154    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-system-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysconfig\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-hostroot\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0632457-1698-4ba2-94de-33bf5e97f4af-tmp-dir\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.052239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-kubernetes\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-systemd\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-etc-tuned\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdghr\" (UniqueName: \"kubernetes.io/projected/28718450-4130-489c-a497-558b4fd200c9-kube-api-access-sdghr\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-socket-dir-parent\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddfd1c58-3ab2-4b8c-b059-a720026747a2-host-slash\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-lib-modules\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-cni-binary-copy\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-conf-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-run\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-tmp\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxwb\" (UniqueName: \"kubernetes.io/projected/35b309ee-baab-4bea-9d9d-e381f60774ea-kube-api-access-wfxwb\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-daemon-config\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.052690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-etc-kubernetes\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddfd1c58-3ab2-4b8c-b059-a720026747a2-iptables-alerter-script\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnz5\" (UniqueName: \"kubernetes.io/projected/f0632457-1698-4ba2-94de-33bf5e97f4af-kube-api-access-jcnz5\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-host\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-bin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-multus\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-var-lib-kubelet\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052887    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-cnibin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-netns\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-modprobe-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.052984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-sys\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2hf\" (UniqueName: \"kubernetes.io/projected/ddfd1c58-3ab2-4b8c-b059-a720026747a2-kube-api-access-qw2hf\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-multus-certs\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053133    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-multus-certs\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-kubelet\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f0632457-1698-4ba2-94de-33bf5e97f4af-hosts-file\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.053418 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053382    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-conf\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053667    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-k8s-cni-cncf-io\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-system-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysconfig\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.053812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-hostroot\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.054115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0632457-1698-4ba2-94de-33bf5e97f4af-tmp-dir\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.054195 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.054189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-kubernetes\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.054510 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.054233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-systemd\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.055023 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.054997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-etc-kubernetes\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.055128 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.055075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-cni-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.055358 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.055339    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:41.055435 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.055414    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.555382792 +0000 UTC m=+3.135310161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:41.055615 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.055587    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-socket-dir-parent\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.055742 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.055689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddfd1c58-3ab2-4b8c-b059-a720026747a2-host-slash\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.055880 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.055863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-lib-modules\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.056360 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-cni-binary-copy\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.056448 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-conf-dir\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.056514 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-run\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.056623 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056605    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ddfd1c58-3ab2-4b8c-b059-a720026747a2-iptables-alerter-script\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-cnibin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056828    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-host\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-sys\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056897    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-bin\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056919    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-sysctl-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056945    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-var-lib-kubelet\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.056925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-run-netns\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.057074    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35b309ee-baab-4bea-9d9d-e381f60774ea-host-var-lib-cni-multus\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.057125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.057107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28718450-4130-489c-a497-558b4fd200c9-etc-modprobe-d\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.057590 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.057180    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35b309ee-baab-4bea-9d9d-e381f60774ea-multus-daemon-config\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.058381 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.058237    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-etc-tuned\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.058904 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.058882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28718450-4130-489c-a497-558b4fd200c9-tmp\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.062771 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.062734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62fs\" (UniqueName: \"kubernetes.io/projected/fad6550a-c538-4e96-9e17-5019310b5341-kube-api-access-j62fs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:41.064975 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.064520    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:41.064975 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.064545    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:41.064975 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.064561    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:41.064975 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.064637    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:51:41.564619197 +0000 UTC m=+3.144546582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:41.067428 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.065462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdghr\" (UniqueName: \"kubernetes.io/projected/28718450-4130-489c-a497-558b4fd200c9-kube-api-access-sdghr\") pod \"tuned-wtfbk\" (UID: \"28718450-4130-489c-a497-558b4fd200c9\") " pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.068068 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.068016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxwb\" (UniqueName: \"kubernetes.io/projected/35b309ee-baab-4bea-9d9d-e381f60774ea-kube-api-access-wfxwb\") pod \"multus-wljxn\" (UID: \"35b309ee-baab-4bea-9d9d-e381f60774ea\") " pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.068230 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.068193    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnz5\" (UniqueName: \"kubernetes.io/projected/f0632457-1698-4ba2-94de-33bf5e97f4af-kube-api-access-jcnz5\") pod \"node-resolver-zrfl6\" (UID: \"f0632457-1698-4ba2-94de-33bf5e97f4af\") " pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.069468 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.069448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2hf\" (UniqueName: \"kubernetes.io/projected/ddfd1c58-3ab2-4b8c-b059-a720026747a2-kube-api-access-qw2hf\") pod \"iptables-alerter-pdr7n\" (UID: \"ddfd1c58-3ab2-4b8c-b059-a720026747a2\") " pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.141008 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.140976    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw"
Sep 10 11:51:41.152047 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.152018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5xr7v"
Sep 10 11:51:41.166760 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.166736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:51:41.173602 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.173583    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:51:41.183204 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.183178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ktlbq"
Sep 10 11:51:41.190910 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.190880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wtfbk"
Sep 10 11:51:41.199720 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.199696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pdr7n"
Sep 10 11:51:41.208601 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.208582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wljxn"
Sep 10 11:51:41.214190 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.214165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zrfl6"
Sep 10 11:51:41.456491 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.456453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:41.456671 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.456592    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:41.456671 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.456666    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:42.456646859 +0000 UTC m=+4.036574236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:41.557628 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.557585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:41.557804 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.557729    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:41.557804 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.557802    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:42.557780583 +0000 UTC m=+4.137707970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:41.658153 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.658121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:41.658356 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.658265    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:41.658356 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.658283    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:41.658356 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.658296    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:41.658474 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.658374    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:51:42.658357637 +0000 UTC m=+4.238285003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:41.891909 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.891856    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:46:40 +0000 UTC" deadline="2028-03-06 21:28:46.736665186 +0000 UTC"
Sep 10 11:51:41.891909 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.891908    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13041h37m4.844761523s"
Sep 10 11:51:41.941289 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:41.941258    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:41.941476 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:41.941418    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:42.147852 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.147826    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0632457_1698_4ba2_94de_33bf5e97f4af.slice/crio-328c94a3d87202888bb2a8911740458bf6f70e8784356035598741a27a8a2efa WatchSource:0}: Error finding container 328c94a3d87202888bb2a8911740458bf6f70e8784356035598741a27a8a2efa: Status 404 returned error can't find the container with id 328c94a3d87202888bb2a8911740458bf6f70e8784356035598741a27a8a2efa
Sep 10 11:51:42.150427 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.149808    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf0ede9_db60_489e_a386_8c46125d0caf.slice/crio-431a7a398019df875fc1de90c58e40a151d303d8269a2a309e5b506e8b4d9c4e WatchSource:0}: Error finding container 431a7a398019df875fc1de90c58e40a151d303d8269a2a309e5b506e8b4d9c4e: Status 404 returned error can't find the container with id 431a7a398019df875fc1de90c58e40a151d303d8269a2a309e5b506e8b4d9c4e
Sep 10 11:51:42.151267 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.151231    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ef79f_b500_41f7_9278_23a45dc7911e.slice/crio-34abc219bc741db9f0dbe8fbce810f391819b6dd9ff615ce5d48a61d7bbee07c WatchSource:0}: Error finding container 34abc219bc741db9f0dbe8fbce810f391819b6dd9ff615ce5d48a61d7bbee07c: Status 404 returned error can't find the container with id 34abc219bc741db9f0dbe8fbce810f391819b6dd9ff615ce5d48a61d7bbee07c
Sep 10 11:51:42.177242 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.177216    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfd1c58_3ab2_4b8c_b059_a720026747a2.slice/crio-3b14141efed23ceba6be60ec1cb3874a749cb081aaeaadb231e4123e9775a67c WatchSource:0}: Error finding container 3b14141efed23ceba6be60ec1cb3874a749cb081aaeaadb231e4123e9775a67c: Status 404 returned error can't find the container with id 3b14141efed23ceba6be60ec1cb3874a749cb081aaeaadb231e4123e9775a67c
Sep 10 11:51:42.177929 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.177903    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b309ee_baab_4bea_9d9d_e381f60774ea.slice/crio-b999f41b7b3e4f01f97465e6c91081be905ce89d89a2b060e18c69fcae0301b9 WatchSource:0}: Error finding container b999f41b7b3e4f01f97465e6c91081be905ce89d89a2b060e18c69fcae0301b9: Status 404 returned error can't find the container with id b999f41b7b3e4f01f97465e6c91081be905ce89d89a2b060e18c69fcae0301b9
Sep 10 11:51:42.178782 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.178759    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aaa613_10e8_4aca_9d8c_7b78a008d92a.slice/crio-b8b72279531f08179efd11997997a30d5132aa617394a5a9bb7bd7591dd26d3d WatchSource:0}: Error finding container b8b72279531f08179efd11997997a30d5132aa617394a5a9bb7bd7591dd26d3d: Status 404 returned error can't find the container with id b8b72279531f08179efd11997997a30d5132aa617394a5a9bb7bd7591dd26d3d
Sep 10 11:51:42.179800 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.179632    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bee52f2_0f1c_4c43_bb6b_0520c65a8af0.slice/crio-3511fa61bcd5199023cdc15c8d0e6e7808efa31bafe21724b9b67db785d59fe7 WatchSource:0}: Error finding container 3511fa61bcd5199023cdc15c8d0e6e7808efa31bafe21724b9b67db785d59fe7: Status 404 returned error can't find the container with id 3511fa61bcd5199023cdc15c8d0e6e7808efa31bafe21724b9b67db785d59fe7
Sep 10 11:51:42.181074 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:51:42.180898    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28149a1e_45ad_42d4_abbf_e26de7a39e82.slice/crio-3e7244e51ef2e5d373ec49879b9e1b3b41bcff0afe2f2a4f76235b190f56b89e WatchSource:0}: Error finding container 3e7244e51ef2e5d373ec49879b9e1b3b41bcff0afe2f2a4f76235b190f56b89e: Status 404 returned error can't find the container with id 3e7244e51ef2e5d373ec49879b9e1b3b41bcff0afe2f2a4f76235b190f56b89e
Sep 10 11:51:42.462821 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.462794    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:42.462952 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.462933    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:42.463003 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.462994    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:44.462979583 +0000 UTC m=+6.042906947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:42.564094 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.564061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:42.564215 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.564168    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:42.564249 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.564220    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:44.564207099 +0000 UTC m=+6.144134463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:42.664998 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.664957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:42.665171 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.665153    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:42.665229 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.665183    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:42.665229 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.665198    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:42.665328 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.665263    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:51:44.665243523 +0000 UTC m=+6.245170902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:42.893160 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.892982    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 11:46:40 +0000 UTC" deadline="2028-06-24 05:56:58.043628478 +0000 UTC"
Sep 10 11:51:42.893160 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.893155    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15666h5m15.150478854s"
Sep 10 11:51:42.943095 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.943068    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:42.943255 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.943198    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:42.944701 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.944498    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:42.944701 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:42.944648    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:42.956163 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.956132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xr7v" event={"ID":"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0","Type":"ContainerStarted","Data":"3511fa61bcd5199023cdc15c8d0e6e7808efa31bafe21724b9b67db785d59fe7"}
Sep 10 11:51:42.961846 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.961822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerStarted","Data":"3e7244e51ef2e5d373ec49879b9e1b3b41bcff0afe2f2a4f76235b190f56b89e"}
Sep 10 11:51:42.974179 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.974093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdr7n" event={"ID":"ddfd1c58-3ab2-4b8c-b059-a720026747a2","Type":"ContainerStarted","Data":"3b14141efed23ceba6be60ec1cb3874a749cb081aaeaadb231e4123e9775a67c"}
Sep 10 11:51:42.977997 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.977892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrfl6" event={"ID":"f0632457-1698-4ba2-94de-33bf5e97f4af","Type":"ContainerStarted","Data":"328c94a3d87202888bb2a8911740458bf6f70e8784356035598741a27a8a2efa"}
Sep 10 11:51:42.981039 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.980546    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal" event={"ID":"7aa9e6d39be639df8387e0cdc021c44c","Type":"ContainerStarted","Data":"8e67d9a89e138353be8a03a560e9f79d54e52c8c9bdf61cc94793fe14e02b898"}
Sep 10 11:51:42.982950 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.982890    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtfn5" event={"ID":"02aaa613-10e8-4aca-9d8c-7b78a008d92a","Type":"ContainerStarted","Data":"b8b72279531f08179efd11997997a30d5132aa617394a5a9bb7bd7591dd26d3d"}
Sep 10 11:51:42.990548 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.990489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wljxn" event={"ID":"35b309ee-baab-4bea-9d9d-e381f60774ea","Type":"ContainerStarted","Data":"b999f41b7b3e4f01f97465e6c91081be905ce89d89a2b060e18c69fcae0301b9"}
Sep 10 11:51:42.994780 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:42.994710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtfbk" event={"ID":"28718450-4130-489c-a497-558b4fd200c9","Type":"ContainerStarted","Data":"b1dd9741eb7c149a75e14efe69fcd13e6edeac10969a3ea93396fe22c566fb56"}
Sep 10 11:51:43.005353 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.005325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw" event={"ID":"1c8ef79f-b500-41f7-9278-23a45dc7911e","Type":"ContainerStarted","Data":"34abc219bc741db9f0dbe8fbce810f391819b6dd9ff615ce5d48a61d7bbee07c"}
Sep 10 11:51:43.021002 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.020975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"431a7a398019df875fc1de90c58e40a151d303d8269a2a309e5b506e8b4d9c4e"}
Sep 10 11:51:43.025318 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.025280    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3b432bc9933fa4742789549900dceff" containerID="8f0d170bf2140341034dd0ed23437700a11843464c1eabf7ca3f86d0057f95a6" exitCode=0
Sep 10 11:51:43.025429 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.025328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal" event={"ID":"f3b432bc9933fa4742789549900dceff","Type":"ContainerDied","Data":"8f0d170bf2140341034dd0ed23437700a11843464c1eabf7ca3f86d0057f95a6"}
Sep 10 11:51:43.042130 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.042078    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-58.ec2.internal" podStartSLOduration=3.042058665 podStartE2EDuration="3.042058665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:42.997454481 +0000 UTC m=+4.577381868" watchObservedRunningTime="2026-09-10 11:51:43.042058665 +0000 UTC m=+4.621986052"
Sep 10 11:51:43.941212 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:43.941179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:43.941735 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:43.941341    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:44.042838 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.042771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal" event={"ID":"f3b432bc9933fa4742789549900dceff","Type":"ContainerStarted","Data":"492adc12e7cef1c1f0c73ff1459c62a805641cbd47a80725c51c31e8054628bf"}
Sep 10 11:51:44.482474 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.482438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:44.482697 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.482606    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:44.482697 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.482677    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:48.482658039 +0000 UTC m=+10.062585417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:44.583016 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.582975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:44.583199 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.583123    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:44.583199 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.583181    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:48.583164564 +0000 UTC m=+10.163091929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:44.684379 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.684337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:44.684639 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.684547    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:44.684639 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.684575    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:44.684639 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.684589    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:44.684837 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.684644    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:51:48.684626062 +0000 UTC m=+10.264553429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:44.943612 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.943576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:44.944065 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.943708    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:44.944128 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:44.944065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:44.944197 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:44.944171    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:45.941213 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:45.941183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:45.941428 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:45.941404    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:46.940921 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:46.940884    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:46.941405 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:46.940887    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:46.941405 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:46.941013    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:46.941405 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:46.941078    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:47.941261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:47.941220    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:47.941654 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:47.941372    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:48.519240 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:48.519107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:48.519439 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.519270    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:48.519439 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.519350    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:56.519331189 +0000 UTC m=+18.099258556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:48.619758 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:48.619727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:48.619930 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.619907    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:48.620001 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.619975    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:51:56.619959946 +0000 UTC m=+18.199887309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:48.720372 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:48.720335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:48.720537 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.720493    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:48.720537 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.720516    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:48.720537 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.720531    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:48.720698 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.720599    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:51:56.72058034 +0000 UTC m=+18.300507707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:48.942097 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:48.942062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:48.942584 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:48.942143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:48.942584 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.942262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:48.942584 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:48.942404    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:49.941363 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:49.941330    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:49.941551 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:49.941454    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:50.943803 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:50.943149    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:50.943803 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:50.943288    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:50.943803 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:50.943375    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:50.943803 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:50.943459    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:51.940854 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:51.940817    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:51.941032 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:51.940945    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:52.941364 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:52.941327    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:52.941714 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:52.941447    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:52.941714 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:52.941498    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:52.941714 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:52.941591    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:53.941116 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:53.941082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:53.941283 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:53.941199    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:54.941135 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:54.941103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:54.941615 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:54.941103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:54.941615 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:54.941232    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:54.941615 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:54.941362    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:55.941017 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:55.940988    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:55.941207 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:55.941103    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:56.576996 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:56.576954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:56.577152 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.577085    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:56.577212 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.577157    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:12.577142142 +0000 UTC m=+34.157069511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:51:56.678151 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:56.678118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:56.678354 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.678284    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:56.678417 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.678373    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:12.678352338 +0000 UTC m=+34.258279702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:51:56.778943 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:56.778907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:56.779099 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.779083    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:51:56.779165 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.779105    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:51:56.779165 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.779116    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:56.779250 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.779166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:52:12.779153396 +0000 UTC m=+34.359080760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:51:56.941212 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:56.941181    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:56.941698 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.941318    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:56.941698 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:56.941383    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:56.941698 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:56.941491    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:57.941049 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:57.941020    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:57.941214 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:57.941131    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:51:58.943958 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:58.943922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:51:58.944408 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:58.944062    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:51:58.944408 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:58.944118    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:51:58.944408 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:58.944238    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:51:59.940874 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:51:59.940833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:51:59.941050 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:51:59.940981    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:00.942103 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:00.940778    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:00.942103 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:00.941136    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:00.942103 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:00.940812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:00.942103 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:00.941246    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:01.080990 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.080784    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wtfbk" event={"ID":"28718450-4130-489c-a497-558b4fd200c9","Type":"ContainerStarted","Data":"6ecaa562a2e62b038da49178d5f4eb2fa121d495e876f66a0a88a7251c994206"}
Sep 10 11:52:01.082634 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.082608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw" event={"ID":"1c8ef79f-b500-41f7-9278-23a45dc7911e","Type":"ContainerStarted","Data":"aa89a047b1e8bb5ed2fe406312ec8c86a7c4b2deadb2237fa9c7eb30df876653"}
Sep 10 11:52:01.085091 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.085063    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"1f98ca6f6b3f57808ec16f83100ec87ec23b7033ebbeb3b193e93d3dfecebc6e"}
Sep 10 11:52:01.085203 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.085145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"796f05d68507b2bc199438f944114188d17a087bf0fc3b632a820f14f47ddccd"}
Sep 10 11:52:01.085203 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.085197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"3f6daedd8d34f271fff1b2e96a2e5da851f049ddffa66de8922dca9c20c72ff3"}
Sep 10 11:52:01.086836 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.086805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5xr7v" event={"ID":"5bee52f2-0f1c-4c43-bb6b-0520c65a8af0","Type":"ContainerStarted","Data":"195d08217805fd99975f1e609d9a7cf3d844de441fee64fe6f6d64ef902ae0cc"}
Sep 10 11:52:01.088607 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.088582    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="df3b51730954a9d54727e900b52f9b1d2de727f9b1cf975d9ad65eec7b55b8af" exitCode=0
Sep 10 11:52:01.088690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.088674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"df3b51730954a9d54727e900b52f9b1d2de727f9b1cf975d9ad65eec7b55b8af"}
Sep 10 11:52:01.090955 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.090713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zrfl6" event={"ID":"f0632457-1698-4ba2-94de-33bf5e97f4af","Type":"ContainerStarted","Data":"1ed54fdb72e7d8e53f7134a53567b6caf7cb3c9941340be99382802cf494047c"}
Sep 10 11:52:01.093661 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.093633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtfn5" event={"ID":"02aaa613-10e8-4aca-9d8c-7b78a008d92a","Type":"ContainerStarted","Data":"7398bf2bd8e52ae044494ab548b46909a140384f6a9cf6dcea529a5bb74d6fc0"}
Sep 10 11:52:01.095139 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.095119    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wljxn" event={"ID":"35b309ee-baab-4bea-9d9d-e381f60774ea","Type":"ContainerStarted","Data":"7e10e117eb6349c7ecac73f6db12f802ab5c750be51c06fad50c5d56742996c3"}
Sep 10 11:52:01.101029 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.100990    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wtfbk" podStartSLOduration=3.605789389 podStartE2EDuration="22.100977192s" totalImagesPullingTime="18.495187803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:01.100474924 +0000 UTC m=+22.680402306" watchObservedRunningTime="2026-09-10 11:52:01.100977192 +0000 UTC m=+22.680904578"
Sep 10 11:52:01.101148 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.101123    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-58.ec2.internal" podStartSLOduration=21.101115377 podStartE2EDuration="21.101115377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:51:44.061268796 +0000 UTC m=+5.641196184" watchObservedRunningTime="2026-09-10 11:52:01.101115377 +0000 UTC m=+22.681042761"
Sep 10 11:52:01.120629 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.120581    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtfn5" podStartSLOduration=7.771445662 podStartE2EDuration="22.1205678s" totalImagesPullingTime="14.349122138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:01.120466106 +0000 UTC m=+22.700393502" watchObservedRunningTime="2026-09-10 11:52:01.1205678 +0000 UTC m=+22.700495186"
Sep 10 11:52:01.138372 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.138322    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zrfl6" podStartSLOduration=3.619399154 podStartE2EDuration="22.138283385s" totalImagesPullingTime="18.518884231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:01.1379308 +0000 UTC m=+22.717858186" watchObservedRunningTime="2026-09-10 11:52:01.138283385 +0000 UTC m=+22.718210770"
Sep 10 11:52:01.218035 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.217917    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wljxn" podStartSLOduration=3.600930216 podStartE2EDuration="22.217900994s" totalImagesPullingTime="18.616970778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:01.200574014 +0000 UTC m=+22.780501401" watchObservedRunningTime="2026-09-10 11:52:01.217900994 +0000 UTC m=+22.797828379"
Sep 10 11:52:01.218255 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.218156    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5xr7v" podStartSLOduration=3.749148999 podStartE2EDuration="22.218144689s" totalImagesPullingTime="18.46899569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:01.217700692 +0000 UTC m=+22.797628276" watchObservedRunningTime="2026-09-10 11:52:01.218144689 +0000 UTC m=+22.798072088"
Sep 10 11:52:01.940990 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.940755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:01.941212 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:01.941051    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:01.995464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.995433    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:52:01.996184 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:01.996161    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:52:02.103240 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.103079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"3b1b1ddd7e9cbe8d5569fdde8d39d956377d5ba8256c019cca795903d0bbdffa"}
Sep 10 11:52:02.103519 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.103379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"3c78636d3f910c69463e8f381fe5f18507c04fd2363eaed870c61b8a26cc0531"}
Sep 10 11:52:02.103519 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.103454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"d322a9ff6cbf07f784cedfbd8a15301043b38dfde53b0577125a115bfa942d34"}
Sep 10 11:52:02.366465 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.366438    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 11:52:02.580720 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.580642    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:52:02.581199 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.581184    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtfn5"
Sep 10 11:52:02.918190 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.918068    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T11:52:02.366458451Z","UUID":"832d4631-be3d-4bca-9dd5-d0416460a12e","Handler":null,"Name":"","Endpoint":""}
Sep 10 11:52:02.919753 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.919730    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 11:52:02.919898 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.919767    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 11:52:02.944675 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.944647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:02.944860 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:02.944647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:02.944860 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:02.944781    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:02.944976 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:02.944864    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:03.107155 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:03.107114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw" event={"ID":"1c8ef79f-b500-41f7-9278-23a45dc7911e","Type":"ContainerStarted","Data":"cc3f96564aa7e1b9bd885aed311ddcd71dc8eeda0c40b7b83f3a3e56b821fde4"}
Sep 10 11:52:03.108892 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:03.108843    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pdr7n" event={"ID":"ddfd1c58-3ab2-4b8c-b059-a720026747a2","Type":"ContainerStarted","Data":"a420f9fe85e1ecfa71b030f9ff733dd71e51ea4ca6b63d99c0b14a84ee09ddef"}
Sep 10 11:52:03.126870 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:03.126820    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pdr7n" podStartSLOduration=5.653105314 podStartE2EDuration="24.126801123s" totalImagesPullingTime="18.473695809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:03.126363583 +0000 UTC m=+24.706290960" watchObservedRunningTime="2026-09-10 11:52:03.126801123 +0000 UTC m=+24.706728510"
Sep 10 11:52:03.941248 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:03.941218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:03.941437 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:03.941367    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:04.113697 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:04.113612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw" event={"ID":"1c8ef79f-b500-41f7-9278-23a45dc7911e","Type":"ContainerStarted","Data":"8362783356ce8585a9a941dc7633fbb56f10297ae0590230cf3f3fe74c80e898"}
Sep 10 11:52:04.116916 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:04.116891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"1f60a85071448f73cf163b68da8b6a30615a9515d7999504acb90adacecfc69b"}
Sep 10 11:52:04.135182 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:04.135138    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bj7bw" podStartSLOduration=4.000300206 podStartE2EDuration="25.135123878s" totalImagesPullingTime="21.134823672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:04.13481118 +0000 UTC m=+25.714738566" watchObservedRunningTime="2026-09-10 11:52:04.135123878 +0000 UTC m=+25.715051264"
Sep 10 11:52:04.944263 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:04.944229    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:04.944451 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:04.944270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:04.944451 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:04.944373    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:04.944562 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:04.944501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:05.941171 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:05.940830    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:05.941171 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:05.941055    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:06.124952 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.124872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" event={"ID":"8bf0ede9-db60-489e-a386-8c46125d0caf","Type":"ContainerStarted","Data":"c560be1c768b8617c3eea487335c1ca3b9540bfa0249bbfcb9f09b181b77089a"}
Sep 10 11:52:06.125243 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.125218    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:06.125360 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.125328    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:06.127114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.127088    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="aaf1becf72d981e988a0b55548afae1484c1e74c089781a5b05412eddf8d6df1" exitCode=0
Sep 10 11:52:06.127222 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.127129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"aaf1becf72d981e988a0b55548afae1484c1e74c089781a5b05412eddf8d6df1"}
Sep 10 11:52:06.141346 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.141322    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:06.141475 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.141462    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:06.161877 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.161835    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2" podStartSLOduration=8.61763038 podStartE2EDuration="27.16182405s" totalImagesPullingTime="18.54419367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:06.161757779 +0000 UTC m=+27.741685345" watchObservedRunningTime="2026-09-10 11:52:06.16182405 +0000 UTC m=+27.741751435"
Sep 10 11:52:06.941293 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.941252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:06.941759 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:06.941272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:06.941759 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:06.941422    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:06.941759 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:06.941563    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:07.129579 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.129554    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 11:52:07.414816 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.414575    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:07.651890 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.651854    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6wq6"]
Sep 10 11:52:07.652112 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.651989    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:07.652112 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:07.652085    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:07.652284 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.652264    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-77jx8"]
Sep 10 11:52:07.652385 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.652372    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:07.652469 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:07.652447    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:07.654036 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.654017    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4brxb"]
Sep 10 11:52:07.654129 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:07.654113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:07.654224 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:07.654205    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:08.133739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:08.133648    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="395d5a912f2607c79f4ee4a540077095057adde9824a5cab84129170a41cdbc5" exitCode=0
Sep 10 11:52:08.134087 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:08.133732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"395d5a912f2607c79f4ee4a540077095057adde9824a5cab84129170a41cdbc5"}
Sep 10 11:52:08.942460 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:08.942432    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:08.942636 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:08.942532    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:08.942715 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:08.942650    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:08.942801 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:08.942777    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:09.941088 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:09.941017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:09.941454 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:09.941112    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:10.140219 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:10.140180    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="948a46153e71f44c5b0a3b290d080e520ef041aa2c48781fe222b245da63c80d" exitCode=0
Sep 10 11:52:10.140408 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:10.140223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"948a46153e71f44c5b0a3b290d080e520ef041aa2c48781fe222b245da63c80d"}
Sep 10 11:52:10.941460 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:10.941428    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:10.941805 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:10.941474    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:10.941805 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:10.941560    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:10.941805 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:10.941651    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:11.940924 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:11.940885    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:11.941108 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:11.941024    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4brxb" podUID="fad6550a-c538-4e96-9e17-5019310b5341"
Sep 10 11:52:12.589980 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:12.589940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:12.590466 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.590072    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 11:52:12.590466 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.590133    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret podName:edc233d2-8734-410f-9add-9b30778726f8 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:44.590113802 +0000 UTC m=+66.170041170 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret") pod "global-pull-secret-syncer-j6wq6" (UID: "edc233d2-8734-410f-9add-9b30778726f8") : object "kube-system"/"original-pull-secret" not registered
Sep 10 11:52:12.691281 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:12.691248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:12.691475 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.691394    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:52:12.691475 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.691460    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:44.691443019 +0000 UTC m=+66.271370400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 11:52:12.792219 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:12.792163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:12.792453 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.792436    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 11:52:12.792518 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.792460    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 11:52:12.792518 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.792474    2573 projected.go:196] Error preparing data for projected volume kube-api-access-7pcf7 for pod openshift-network-diagnostics/network-check-target-77jx8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:52:12.792609 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.792537    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7 podName:73c84e9f-5cfe-4541-8d85-da72eb00234d nodeName:}" failed. No retries permitted until 2026-09-10 11:52:44.79251981 +0000 UTC m=+66.372447188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7pcf7" (UniqueName: "kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7") pod "network-check-target-77jx8" (UID: "73c84e9f-5cfe-4541-8d85-da72eb00234d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 11:52:12.941202 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:12.941171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:12.941378 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:12.941173    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:12.941378 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.941282    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j6wq6" podUID="edc233d2-8734-410f-9add-9b30778726f8"
Sep 10 11:52:12.941455 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:12.941407    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-77jx8" podUID="73c84e9f-5cfe-4541-8d85-da72eb00234d"
Sep 10 11:52:13.693389 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.693151    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-58.ec2.internal" event="NodeReady"
Sep 10 11:52:13.693766 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.693513    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 11:52:13.744836 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.744801    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cgs5d"]
Sep 10 11:52:13.747491 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.747470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:13.750421 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.750398    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 11:52:13.750569 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.750545    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 11:52:13.750647 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.750605    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q76h9\""
Sep 10 11:52:13.752023 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.752003    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hxmcd"]
Sep 10 11:52:13.753804 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.753783    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:13.757975 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.757954    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 11:52:13.758081 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.757954    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2c8qf\""
Sep 10 11:52:13.758175 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.758130    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 11:52:13.758461 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.758444    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 11:52:13.761165 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.761144    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgs5d"]
Sep 10 11:52:13.767606 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.767589    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxmcd"]
Sep 10 11:52:13.900557 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb874de1-6601-4984-9f35-f0a9c30e9d3d-cert\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:13.900716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4t7\" (UniqueName: \"kubernetes.io/projected/a486eacf-3b31-4de8-9a5d-fcbff00af988-kube-api-access-tc4t7\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:13.900716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:13.900716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900703    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdf7\" (UniqueName: \"kubernetes.io/projected/cb874de1-6601-4984-9f35-f0a9c30e9d3d-kube-api-access-fcdf7\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:13.900851 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a486eacf-3b31-4de8-9a5d-fcbff00af988-tmp-dir\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:13.900851 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.900769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a486eacf-3b31-4de8-9a5d-fcbff00af988-config-volume\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:13.940728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.940669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:13.943767 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.943735    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrvwr\""
Sep 10 11:52:13.943894 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:13.943832    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 11:52:14.001544 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a486eacf-3b31-4de8-9a5d-fcbff00af988-tmp-dir\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.001674 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a486eacf-3b31-4de8-9a5d-fcbff00af988-config-volume\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.001674 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb874de1-6601-4984-9f35-f0a9c30e9d3d-cert\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:14.001780 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4t7\" (UniqueName: \"kubernetes.io/projected/a486eacf-3b31-4de8-9a5d-fcbff00af988-kube-api-access-tc4t7\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.001780 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.001877 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdf7\" (UniqueName: \"kubernetes.io/projected/cb874de1-6601-4984-9f35-f0a9c30e9d3d-kube-api-access-fcdf7\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:14.001925 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.001899    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a486eacf-3b31-4de8-9a5d-fcbff00af988-tmp-dir\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.001967 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:14.001953    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:14.002046 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:14.002032    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:14.502008906 +0000 UTC m=+36.081936273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:14.002191 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.002174    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a486eacf-3b31-4de8-9a5d-fcbff00af988-config-volume\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.006282 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.006262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb874de1-6601-4984-9f35-f0a9c30e9d3d-cert\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:14.011336 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.011289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdf7\" (UniqueName: \"kubernetes.io/projected/cb874de1-6601-4984-9f35-f0a9c30e9d3d-kube-api-access-fcdf7\") pod \"ingress-canary-hxmcd\" (UID: \"cb874de1-6601-4984-9f35-f0a9c30e9d3d\") " pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:14.011445 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.011397    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4t7\" (UniqueName: \"kubernetes.io/projected/a486eacf-3b31-4de8-9a5d-fcbff00af988-kube-api-access-tc4t7\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.066082 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.066055    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxmcd"
Sep 10 11:52:14.505753 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.505728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:14.505865 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:14.505839    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:14.505900 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:14.505884    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:15.50587167 +0000 UTC m=+37.085799033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:14.703739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.703680    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxmcd"]
Sep 10 11:52:14.706816 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:14.706790    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb874de1_6601_4984_9f35_f0a9c30e9d3d.slice/crio-23232ab479a220c49335b1f941dd0192a9f1364d67104c69f603365c640e9f26 WatchSource:0}: Error finding container 23232ab479a220c49335b1f941dd0192a9f1364d67104c69f603365c640e9f26: Status 404 returned error can't find the container with id 23232ab479a220c49335b1f941dd0192a9f1364d67104c69f603365c640e9f26
Sep 10 11:52:14.940554 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.940526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:14.940674 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.940537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:14.944339 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.943775    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 11:52:14.945042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.944677    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 11:52:14.945042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.944774    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69fjj\""
Sep 10 11:52:14.945042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:14.944935    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 11:52:15.154784 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:15.154745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxmcd" event={"ID":"cb874de1-6601-4984-9f35-f0a9c30e9d3d","Type":"ContainerStarted","Data":"23232ab479a220c49335b1f941dd0192a9f1364d67104c69f603365c640e9f26"}
Sep 10 11:52:15.158127 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:15.158089    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="1af0f0927ab9a055c8b05cd7dae63d294b555fdaccd0526bc5d5fab45ce391e4" exitCode=0
Sep 10 11:52:15.158266 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:15.158151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"1af0f0927ab9a055c8b05cd7dae63d294b555fdaccd0526bc5d5fab45ce391e4"}
Sep 10 11:52:15.514112 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:15.514031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:15.514255 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:15.514190    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:15.514294 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:15.514259    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:17.514240455 +0000 UTC m=+39.094167825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:16.164612 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:16.164566    2573 generic.go:356] "Generic (PLEG): container finished" podID="28149a1e-45ad-42d4-abbf-e26de7a39e82" containerID="239b25d6a99a3cc6a8c12dbc4f178962570ea0d86e0e2979da428fd9d5e54e5a" exitCode=0
Sep 10 11:52:16.165066 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:16.164624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerDied","Data":"239b25d6a99a3cc6a8c12dbc4f178962570ea0d86e0e2979da428fd9d5e54e5a"}
Sep 10 11:52:17.169269 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:17.169230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxmcd" event={"ID":"cb874de1-6601-4984-9f35-f0a9c30e9d3d","Type":"ContainerStarted","Data":"1868021ea9d7a57eb9355ac75ad6fa23d00dd3da21fa6ace0839de7a714ab5ed"}
Sep 10 11:52:17.172891 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:17.172860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" event={"ID":"28149a1e-45ad-42d4-abbf-e26de7a39e82","Type":"ContainerStarted","Data":"6093267dd7df03e9059038d2a37def857fa81693e20bc33f65bc449b1263de46"}
Sep 10 11:52:17.190131 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:17.190073    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hxmcd" podStartSLOduration=1.9468215180000001 podStartE2EDuration="4.190051958s" totalImagesPullingTime="2.24323044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:17.189470447 +0000 UTC m=+38.769397836" watchObservedRunningTime="2026-09-10 11:52:17.190051958 +0000 UTC m=+38.769979345"
Sep 10 11:52:17.225420 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:17.225373    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ktlbq" podStartSLOduration=8.638395879 podStartE2EDuration="38.225356419s" totalImagesPullingTime="29.58696054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:17.224404609 +0000 UTC m=+38.804331995" watchObservedRunningTime="2026-09-10 11:52:17.225356419 +0000 UTC m=+38.805283804"
Sep 10 11:52:17.530213 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:17.530179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:17.530383 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:17.530290    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:17.530383 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:17.530365    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:21.530351028 +0000 UTC m=+43.110278392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:21.556087 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:21.556053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:21.556506 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:21.556173    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:21.556506 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:21.556224    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:29.556210908 +0000 UTC m=+51.136138273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:29.610545 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:29.610504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:29.610982 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:29.610627    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:29.610982 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:29.610694    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:45.610679829 +0000 UTC m=+67.190607193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:39.157721 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:39.157686    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4nhr2"
Sep 10 11:52:44.618805 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.618749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:44.621966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.621949    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 11:52:44.633017 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.632997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/edc233d2-8734-410f-9add-9b30778726f8-original-pull-secret\") pod \"global-pull-secret-syncer-j6wq6\" (UID: \"edc233d2-8734-410f-9add-9b30778726f8\") " pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:44.655224 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.655193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6wq6"
Sep 10 11:52:44.721008 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.720821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:52:44.724095 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.724074    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 11:52:44.731510 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:44.731493    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 11:52:44.731601 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:44.731556    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs podName:fad6550a-c538-4e96-9e17-5019310b5341 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:48.731534457 +0000 UTC m=+130.311461839 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs") pod "network-metrics-daemon-4brxb" (UID: "fad6550a-c538-4e96-9e17-5019310b5341") : secret "metrics-daemon-secret" not found
Sep 10 11:52:44.821972 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.821935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:44.825236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.825214    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 11:52:44.835209 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.835185    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 11:52:44.846036 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.846016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcf7\" (UniqueName: \"kubernetes.io/projected/73c84e9f-5cfe-4541-8d85-da72eb00234d-kube-api-access-7pcf7\") pod \"network-check-target-77jx8\" (UID: \"73c84e9f-5cfe-4541-8d85-da72eb00234d\") " pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:44.925285 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.925221    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6wq6"]
Sep 10 11:52:44.928031 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:44.928009    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc233d2_8734_410f_9add_9b30778726f8.slice/crio-2b8cd846eccd67f9d32212abb027eaf471404a4b94918b7ddc1f037370f75782 WatchSource:0}: Error finding container 2b8cd846eccd67f9d32212abb027eaf471404a4b94918b7ddc1f037370f75782: Status 404 returned error can't find the container with id 2b8cd846eccd67f9d32212abb027eaf471404a4b94918b7ddc1f037370f75782
Sep 10 11:52:44.953509 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.953491    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69fjj\""
Sep 10 11:52:44.960992 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:44.960960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:45.239244 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.239175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6wq6" event={"ID":"edc233d2-8734-410f-9add-9b30778726f8","Type":"ContainerStarted","Data":"2b8cd846eccd67f9d32212abb027eaf471404a4b94918b7ddc1f037370f75782"}
Sep 10 11:52:45.255824 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.255796    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-77jx8"]
Sep 10 11:52:45.258877 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:45.258854    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c84e9f_5cfe_4541_8d85_da72eb00234d.slice/crio-8a07a9159a61d7d25d815097de828d22055561a38901f6c4f046ca38635299c0 WatchSource:0}: Error finding container 8a07a9159a61d7d25d815097de828d22055561a38901f6c4f046ca38635299c0: Status 404 returned error can't find the container with id 8a07a9159a61d7d25d815097de828d22055561a38901f6c4f046ca38635299c0
Sep 10 11:52:45.618537 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.618466    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"]
Sep 10 11:52:45.628385 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.628244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:52:45.628805 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:45.628389    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 11:52:45.628805 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:45.628456    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls podName:a486eacf-3b31-4de8-9a5d-fcbff00af988 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:17.628436828 +0000 UTC m=+99.208364211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls") pod "dns-default-cgs5d" (UID: "a486eacf-3b31-4de8-9a5d-fcbff00af988") : secret "dns-default-metrics-tls" not found
Sep 10 11:52:45.631118 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.631093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.634114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.634096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 11:52:45.634114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.634105    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:52:45.634674 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.634652    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 11:52:45.634918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.634904    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 11:52:45.635450 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.635432    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2h9w7\""
Sep 10 11:52:45.641507 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.641480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"]
Sep 10 11:52:45.729542 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.729506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzl2q\" (UniqueName: \"kubernetes.io/projected/a2c142af-6a36-4984-ba8f-8255afadd1e4-kube-api-access-vzl2q\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.729700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.729589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c142af-6a36-4984-ba8f-8255afadd1e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.729700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.729612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c142af-6a36-4984-ba8f-8255afadd1e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.746707 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.746676    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hwhk6"]
Sep 10 11:52:45.759465 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.759440    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.762395 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.762374    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 11:52:45.762760 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.762728    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:52:45.763075 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.763055    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 11:52:45.763177 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.763147    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 11:52:45.764370 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.764330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hwhk6"]
Sep 10 11:52:45.765740 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.765102    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ndx2q\""
Sep 10 11:52:45.769270 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.769250    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 11:52:45.828709 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.828678    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jnmld"]
Sep 10 11:52:45.830801 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c142af-6a36-4984-ba8f-8255afadd1e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.830918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-serving-cert\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.830918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830832    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c142af-6a36-4984-ba8f-8255afadd1e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.830918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzl2q\" (UniqueName: \"kubernetes.io/projected/a2c142af-6a36-4984-ba8f-8255afadd1e4-kube-api-access-vzl2q\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.830918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-config\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.830918 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdlt\" (UniqueName: \"kubernetes.io/projected/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-kube-api-access-jsdlt\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.831177 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.830979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-trusted-ca\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.831445 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.831421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2c142af-6a36-4984-ba8f-8255afadd1e4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.834125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.834104    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c142af-6a36-4984-ba8f-8255afadd1e4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.847212 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.847193    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54478f5d74-kjsrv"]
Sep 10 11:52:45.848145 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.848129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld"
Sep 10 11:52:45.856962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.856942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cmhlf\""
Sep 10 11:52:45.859034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.858795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzl2q\" (UniqueName: \"kubernetes.io/projected/a2c142af-6a36-4984-ba8f-8255afadd1e4-kube-api-access-vzl2q\") pod \"kube-storage-version-migrator-operator-8d7dd4675-hr2b2\" (UID: \"a2c142af-6a36-4984-ba8f-8255afadd1e4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:45.873491 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.873437    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jnmld"]
Sep 10 11:52:45.873491 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.873473    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54478f5d74-kjsrv"]
Sep 10 11:52:45.873642 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.873574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.877368 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.877342    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lnr4m\""
Sep 10 11:52:45.877680 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.877664    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 11:52:45.877936 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.877921    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 11:52:45.878187 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.878169    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 11:52:45.878501 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.878485    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 11:52:45.879675 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.879656    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 11:52:45.880132 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.880111    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 11:52:45.931318 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-config\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.931447 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.931447 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931400    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdlt\" (UniqueName: \"kubernetes.io/projected/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-kube-api-access-jsdlt\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.931670 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-stats-auth\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.931833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrtq\" (UniqueName: \"kubernetes.io/projected/8ff360af-f86b-463f-94d2-d53c61a410bd-kube-api-access-xlrtq\") pod \"network-check-source-685456655f-jnmld\" (UID: \"8ff360af-f86b-463f-94d2-d53c61a410bd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld"
Sep 10 11:52:45.931833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-trusted-ca\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.931833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dx2\" (UniqueName: \"kubernetes.io/projected/0cb09d4e-faf7-49d9-81e6-1d7504d65425-kube-api-access-n5dx2\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.931833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.931833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-serving-cert\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.932081 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-default-certificate\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:45.932081 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.931937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-config\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.932625 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.932605    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-trusted-ca\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.934668 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.934649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-serving-cert\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.942357 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.942333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdlt\" (UniqueName: \"kubernetes.io/projected/7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc-kube-api-access-jsdlt\") pod \"console-operator-85b454874b-hwhk6\" (UID: \"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc\") " pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:45.949070 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:45.949049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dx2\" (UniqueName: \"kubernetes.io/projected/0cb09d4e-faf7-49d9-81e6-1d7504d65425-kube-api-access-n5dx2\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-default-certificate\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-stats-auth\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.033345    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrtq\" (UniqueName: \"kubernetes.io/projected/8ff360af-f86b-463f-94d2-d53c61a410bd-kube-api-access-xlrtq\") pod \"network-check-source-685456655f-jnmld\" (UID: \"8ff360af-f86b-463f-94d2-d53c61a410bd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld"
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.033763    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.033836    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:46.533816264 +0000 UTC m=+68.113743630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : secret "router-metrics-certs-default" not found
Sep 10 11:52:46.034185 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.034143    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:46.53412356 +0000 UTC m=+68.114050931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:52:46.036427 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.036391    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-stats-auth\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.036611 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.036579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-default-certificate\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.045802 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.045758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrtq\" (UniqueName: \"kubernetes.io/projected/8ff360af-f86b-463f-94d2-d53c61a410bd-kube-api-access-xlrtq\") pod \"network-check-source-685456655f-jnmld\" (UID: \"8ff360af-f86b-463f-94d2-d53c61a410bd\") " pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld"
Sep 10 11:52:46.047783 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.047762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dx2\" (UniqueName: \"kubernetes.io/projected/0cb09d4e-faf7-49d9-81e6-1d7504d65425-kube-api-access-n5dx2\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.071568 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.071546    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:46.158885 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.158701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld"
Sep 10 11:52:46.243175 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.243063    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2"]
Sep 10 11:52:46.246614 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.246533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-77jx8" event={"ID":"73c84e9f-5cfe-4541-8d85-da72eb00234d","Type":"ContainerStarted","Data":"8a07a9159a61d7d25d815097de828d22055561a38901f6c4f046ca38635299c0"}
Sep 10 11:52:46.248530 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:46.248500    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c142af_6a36_4984_ba8f_8255afadd1e4.slice/crio-4cf5095dd7dcc3c78f191e38098c6d60e22b8873322a652a70a237f9d0ca9aac WatchSource:0}: Error finding container 4cf5095dd7dcc3c78f191e38098c6d60e22b8873322a652a70a237f9d0ca9aac: Status 404 returned error can't find the container with id 4cf5095dd7dcc3c78f191e38098c6d60e22b8873322a652a70a237f9d0ca9aac
Sep 10 11:52:46.393719 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.392862    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hwhk6"]
Sep 10 11:52:46.397078 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:46.397047    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bca6299_3fa1_4da5_9bc7_5fd01d13a4fc.slice/crio-5a6c37403e45e154e801c5b3bc12c1a757fd5aa96231044a3be995e858cedeab WatchSource:0}: Error finding container 5a6c37403e45e154e801c5b3bc12c1a757fd5aa96231044a3be995e858cedeab: Status 404 returned error can't find the container with id 5a6c37403e45e154e801c5b3bc12c1a757fd5aa96231044a3be995e858cedeab
Sep 10 11:52:46.463116 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.463031    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-jnmld"]
Sep 10 11:52:46.468834 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:46.468807    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff360af_f86b_463f_94d2_d53c61a410bd.slice/crio-57fdf901dd32dd5236a52f68363732690f92e0fe698230c2cb881fc7b2682146 WatchSource:0}: Error finding container 57fdf901dd32dd5236a52f68363732690f92e0fe698230c2cb881fc7b2682146: Status 404 returned error can't find the container with id 57fdf901dd32dd5236a52f68363732690f92e0fe698230c2cb881fc7b2682146
Sep 10 11:52:46.537478 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.537437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.537618 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:46.537546    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:46.537717 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.537694    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:47.537673968 +0000 UTC m=+69.117601336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:52:46.538132 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.538118    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 11:52:46.538188 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:46.538170    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:47.538154133 +0000 UTC m=+69.118081499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : secret "router-metrics-certs-default" not found
Sep 10 11:52:47.252596 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:47.252557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2" event={"ID":"a2c142af-6a36-4984-ba8f-8255afadd1e4","Type":"ContainerStarted","Data":"4cf5095dd7dcc3c78f191e38098c6d60e22b8873322a652a70a237f9d0ca9aac"}
Sep 10 11:52:47.256397 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:47.256372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld" event={"ID":"8ff360af-f86b-463f-94d2-d53c61a410bd","Type":"ContainerStarted","Data":"57fdf901dd32dd5236a52f68363732690f92e0fe698230c2cb881fc7b2682146"}
Sep 10 11:52:47.259562 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:47.259536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" event={"ID":"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc","Type":"ContainerStarted","Data":"5a6c37403e45e154e801c5b3bc12c1a757fd5aa96231044a3be995e858cedeab"}
Sep 10 11:52:47.547549 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:47.547452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:47.547549 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:47.547532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:47.547775 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:47.547652    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:49.547638055 +0000 UTC m=+71.127565419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:52:47.547775 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:47.547723    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 11:52:47.547775 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:47.547747    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:49.547739284 +0000 UTC m=+71.127666648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : secret "router-metrics-certs-default" not found
Sep 10 11:52:49.562316 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:49.562264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:49.562800 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:49.562354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:49.562800 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:49.562438    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:53.562418034 +0000 UTC m=+75.142345422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:52:49.562800 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:49.562528    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 11:52:49.562800 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:49.562591    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:53.562572157 +0000 UTC m=+75.142499525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : secret "router-metrics-certs-default" not found
Sep 10 11:52:50.411975 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.411940    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nwhf7"]
Sep 10 11:52:50.419007 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.418979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.422118 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.422092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 11:52:50.422252 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.422212    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 11:52:50.422252 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.422224    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 11:52:50.423406 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.423383    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s4qzc\""
Sep 10 11:52:50.423503 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.423419    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 11:52:50.430767 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.430743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwhf7"]
Sep 10 11:52:50.468754 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.468720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.468919 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.468792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8be26753-c6b2-4a1f-a7e9-9beb72381360-data-volume\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.468919 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.468882    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.469041 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.468945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8be26753-c6b2-4a1f-a7e9-9beb72381360-crio-socket\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.469041 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.468983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgcp5\" (UniqueName: \"kubernetes.io/projected/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-api-access-lgcp5\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570176 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8be26753-c6b2-4a1f-a7e9-9beb72381360-data-volume\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8be26753-c6b2-4a1f-a7e9-9beb72381360-crio-socket\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:50.570275    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcp5\" (UniqueName: \"kubernetes.io/projected/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-api-access-lgcp5\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:50.570369    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:51.070347674 +0000 UTC m=+72.650275040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : configmap "kube-rbac-proxy" not found
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:50.570402    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:50.570460    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:51.070441865 +0000 UTC m=+72.650369231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8be26753-c6b2-4a1f-a7e9-9beb72381360-crio-socket\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.570648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.570558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8be26753-c6b2-4a1f-a7e9-9beb72381360-data-volume\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:50.580216 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:50.580196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcp5\" (UniqueName: \"kubernetes.io/projected/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-api-access-lgcp5\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:51.075369 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:51.075330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:51.075556 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:51.075446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:51.075556 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:51.075495    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:51.075556 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:51.075543    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:52:51.075687 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:51.075570    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:52.075554028 +0000 UTC m=+73.655481393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:51.075687 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:51.075594    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:52.075577826 +0000 UTC m=+73.655505191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : configmap "kube-rbac-proxy" not found
Sep 10 11:52:52.083906 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.083877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:52.084275 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.083929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:52.084275 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:52.084042    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:52.084275 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:52.084095    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:54.084075975 +0000 UTC m=+75.664003356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:52.084275 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:52.084040    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:52:52.084275 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:52.084138    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:54.084126589 +0000 UTC m=+75.664053980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : configmap "kube-rbac-proxy" not found
Sep 10 11:52:52.276663 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.276628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld" event={"ID":"8ff360af-f86b-463f-94d2-d53c61a410bd","Type":"ContainerStarted","Data":"32e1b9207978ed80c7b12ad91c137aaaab22dffc65bbb259cd4154a23ae3c05b"}
Sep 10 11:52:52.278371 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.278349    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/0.log"
Sep 10 11:52:52.278507 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.278386    2573 generic.go:356] "Generic (PLEG): container finished" podID="7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc" containerID="567a3092e61129c1254b79bda1ec0eb7a028a19fb06c124d43173307aa821389" exitCode=255
Sep 10 11:52:52.278507 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.278447    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" event={"ID":"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc","Type":"ContainerDied","Data":"567a3092e61129c1254b79bda1ec0eb7a028a19fb06c124d43173307aa821389"}
Sep 10 11:52:52.278690 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.278661    2573 scope.go:117] "RemoveContainer" containerID="567a3092e61129c1254b79bda1ec0eb7a028a19fb06c124d43173307aa821389"
Sep 10 11:52:52.280085 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.280061    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6wq6" event={"ID":"edc233d2-8734-410f-9add-9b30778726f8","Type":"ContainerStarted","Data":"6c7b62110e8a1aa25bbfc94f22f5c34ed14eb188f8939c6a9763fd50edb572fe"}
Sep 10 11:52:52.281912 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.281875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-77jx8" event={"ID":"73c84e9f-5cfe-4541-8d85-da72eb00234d","Type":"ContainerStarted","Data":"af4ba4156953b1599412e637c2b7a1b0602c1b7ab028078d6f553823af670285"}
Sep 10 11:52:52.282016 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.281986    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:52:52.283570 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.283545    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2" event={"ID":"a2c142af-6a36-4984-ba8f-8255afadd1e4","Type":"ContainerStarted","Data":"4941b563f78314a56539f0a06504f3fa04fe845574f43aa84c03f22b209875ca"}
Sep 10 11:52:52.296919 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.296881    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-jnmld" podStartSLOduration=2.310498828 podStartE2EDuration="7.296869907s" totalImagesPullingTime="4.986371079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:52.295670285 +0000 UTC m=+73.875597672" watchObservedRunningTime="2026-09-10 11:52:52.296869907 +0000 UTC m=+73.876797286"
Sep 10 11:52:52.332959 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.332918    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2" podStartSLOduration=2.12294226 podStartE2EDuration="7.332904835s" totalImagesPullingTime="5.209962575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:52.331878966 +0000 UTC m=+73.911806353" watchObservedRunningTime="2026-09-10 11:52:52.332904835 +0000 UTC m=+73.912832220"
Sep 10 11:52:52.358245 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.358187    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j6wq6" podStartSLOduration=66.809914857 podStartE2EDuration="1m13.358165295s" totalImagesPullingTime="6.548250438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:52.356741046 +0000 UTC m=+73.936668429" watchObservedRunningTime="2026-09-10 11:52:52.358165295 +0000 UTC m=+73.938092682"
Sep 10 11:52:52.541891 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.541861    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrfl6_f0632457-1698-4ba2-94de-33bf5e97f4af/dns-node-resolver/0.log"
Sep 10 11:52:52.794257 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.793995    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-77jx8" podStartSLOduration=67.59003449 podStartE2EDuration="1m13.793976737s" totalImagesPullingTime="6.203942247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:52.374611773 +0000 UTC m=+73.954539154" watchObservedRunningTime="2026-09-10 11:52:52.793976737 +0000 UTC m=+74.373904124"
Sep 10 11:52:52.794757 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.794736    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-shftf"]
Sep 10 11:52:52.796926 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.796906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.799593 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.799572    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 11:52:52.799717 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.799694    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 11:52:52.799717 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.799698    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 11:52:52.800753 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.800727    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lxxbn\""
Sep 10 11:52:52.801000 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.800857    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 11:52:52.807773 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.807756    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-shftf"]
Sep 10 11:52:52.890726 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.890517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-key\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.890726 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.890578    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfb5p\" (UniqueName: \"kubernetes.io/projected/681b05bb-c15b-42bb-a153-f2698ad4b35b-kube-api-access-lfb5p\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.890726 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.890611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-cabundle\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.991904 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.991870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-cabundle\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.992055 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.991974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-key\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.992055 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.992002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfb5p\" (UniqueName: \"kubernetes.io/projected/681b05bb-c15b-42bb-a153-f2698ad4b35b-kube-api-access-lfb5p\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.992547 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.992517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-cabundle\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:52.994469 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:52.994446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/681b05bb-c15b-42bb-a153-f2698ad4b35b-signing-key\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:53.000908 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.000885    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfb5p\" (UniqueName: \"kubernetes.io/projected/681b05bb-c15b-42bb-a153-f2698ad4b35b-kube-api-access-lfb5p\") pod \"service-ca-75d99c484-shftf\" (UID: \"681b05bb-c15b-42bb-a153-f2698ad4b35b\") " pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:53.105435 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.105324    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-shftf"
Sep 10 11:52:53.144374 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.144345    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xr7v_5bee52f2-0f1c-4c43-bb6b-0520c65a8af0/node-ca/0.log"
Sep 10 11:52:53.288573 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.288547    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 11:52:53.288962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.288948    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/0.log"
Sep 10 11:52:53.289006 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.288981    2573 generic.go:356] "Generic (PLEG): container finished" podID="7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc" containerID="8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5" exitCode=255
Sep 10 11:52:53.289101 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.289078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" event={"ID":"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc","Type":"ContainerDied","Data":"8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5"}
Sep 10 11:52:53.289153 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.289142    2573 scope.go:117] "RemoveContainer" containerID="567a3092e61129c1254b79bda1ec0eb7a028a19fb06c124d43173307aa821389"
Sep 10 11:52:53.289367 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.289345    2573 scope.go:117] "RemoveContainer" containerID="8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5"
Sep 10 11:52:53.289547 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:53.289525    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hwhk6_openshift-console-operator(7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc)\"" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" podUID="7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc"
Sep 10 11:52:53.386251 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.386229    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-shftf"]
Sep 10 11:52:53.388481 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:52:53.388450    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681b05bb_c15b_42bb_a153_f2698ad4b35b.slice/crio-33aad3d5cd700dcccb899cdf181ae568672188830a568dd83b9471b10c184f96 WatchSource:0}: Error finding container 33aad3d5cd700dcccb899cdf181ae568672188830a568dd83b9471b10c184f96: Status 404 returned error can't find the container with id 33aad3d5cd700dcccb899cdf181ae568672188830a568dd83b9471b10c184f96
Sep 10 11:52:53.597945 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.597909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:53.598117 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:53.597996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:52:53.598117 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:53.598056    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 11:52:53.598202 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:53.598136    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:01.598123088 +0000 UTC m=+83.178050465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:52:53.598202 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:53.598153    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:01.598146714 +0000 UTC m=+83.178074077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : secret "router-metrics-certs-default" not found
Sep 10 11:52:54.102438 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.102399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:54.102630 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.102488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:54.102630 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:54.102530    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:52:54.102630 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:54.102615    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:58.102591272 +0000 UTC m=+79.682518648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : configmap "kube-rbac-proxy" not found
Sep 10 11:52:54.102752 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:54.102656    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:54.102752 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:54.102720    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:52:58.102699723 +0000 UTC m=+79.682627092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:54.153433 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.153399    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hxmcd_cb874de1-6601-4984-9f35-f0a9c30e9d3d/serve-healthcheck-canary/0.log"
Sep 10 11:52:54.293443 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.293407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-shftf" event={"ID":"681b05bb-c15b-42bb-a153-f2698ad4b35b","Type":"ContainerStarted","Data":"33aad3d5cd700dcccb899cdf181ae568672188830a568dd83b9471b10c184f96"}
Sep 10 11:52:54.295102 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.295078    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 11:52:54.295493 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:54.295473    2573 scope.go:117] "RemoveContainer" containerID="8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5"
Sep 10 11:52:54.295644 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:54.295624    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hwhk6_openshift-console-operator(7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc)\"" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" podUID="7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc"
Sep 10 11:52:55.300388 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:55.300320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-shftf" event={"ID":"681b05bb-c15b-42bb-a153-f2698ad4b35b","Type":"ContainerStarted","Data":"934b860036d0012c448e506c5a519ae3c892457fc2756f6ec97cf891c88fc49b"}
Sep 10 11:52:55.318068 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:55.318021    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-shftf" podStartSLOduration=1.465552716 podStartE2EDuration="3.318006664s" totalImagesPullingTime="1.852453948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:52:55.316210245 +0000 UTC m=+76.896137631" watchObservedRunningTime="2026-09-10 11:52:55.318006664 +0000 UTC m=+76.897934049"
Sep 10 11:52:56.071884 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:56.071851    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:56.072026 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:56.071904    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:52:56.072201 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:56.072190    2573 scope.go:117] "RemoveContainer" containerID="8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5"
Sep 10 11:52:56.072362 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:56.072344    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hwhk6_openshift-console-operator(7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc)\"" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" podUID="7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc"
Sep 10 11:52:58.133518 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:58.133474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:58.133976 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:52:58.133565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:52:58.133976 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:58.133636    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 11:52:58.133976 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:58.133649    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 11:52:58.133976 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:58.133696    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:06.133682988 +0000 UTC m=+87.713610351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : configmap "kube-rbac-proxy" not found
Sep 10 11:52:58.133976 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:52:58.133731    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls podName:8be26753-c6b2-4a1f-a7e9-9beb72381360 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:06.133713936 +0000 UTC m=+87.713641327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwhf7" (UID: "8be26753-c6b2-4a1f-a7e9-9beb72381360") : secret "insights-runtime-extractor-tls" not found
Sep 10 11:53:01.662989 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:01.662853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:01.662989 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:01.662897    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:01.663610 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:53:01.663044    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle podName:0cb09d4e-faf7-49d9-81e6-1d7504d65425 nodeName:}" failed. No retries permitted until 2026-09-10 11:53:17.663022038 +0000 UTC m=+99.242949406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle") pod "router-default-54478f5d74-kjsrv" (UID: "0cb09d4e-faf7-49d9-81e6-1d7504d65425") : configmap references non-existent config key: service-ca.crt
Sep 10 11:53:01.665523 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:01.665502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb09d4e-faf7-49d9-81e6-1d7504d65425-metrics-certs\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:06.200254 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.200215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:53:06.200707 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.200282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:53:06.200780 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.200743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8be26753-c6b2-4a1f-a7e9-9beb72381360-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:53:06.202610 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.202589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8be26753-c6b2-4a1f-a7e9-9beb72381360-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwhf7\" (UID: \"8be26753-c6b2-4a1f-a7e9-9beb72381360\") " pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:53:06.329799 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.329766    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwhf7"
Sep 10 11:53:06.604105 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:06.604040    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwhf7"]
Sep 10 11:53:06.607092 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:06.607062    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be26753_c6b2_4a1f_a7e9_9beb72381360.slice/crio-f78841e1de5463baa4a0332aea6b3ee2fe447f9a74066fd26e435c7ee5baabcf WatchSource:0}: Error finding container f78841e1de5463baa4a0332aea6b3ee2fe447f9a74066fd26e435c7ee5baabcf: Status 404 returned error can't find the container with id f78841e1de5463baa4a0332aea6b3ee2fe447f9a74066fd26e435c7ee5baabcf
Sep 10 11:53:07.343202 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:07.343169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwhf7" event={"ID":"8be26753-c6b2-4a1f-a7e9-9beb72381360","Type":"ContainerStarted","Data":"46f1081cfaba9fe8cc910839a5d439ff50e748e771f1803745eb85adf5c0bfde"}
Sep 10 11:53:07.343615 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:07.343227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwhf7" event={"ID":"8be26753-c6b2-4a1f-a7e9-9beb72381360","Type":"ContainerStarted","Data":"f78841e1de5463baa4a0332aea6b3ee2fe447f9a74066fd26e435c7ee5baabcf"}
Sep 10 11:53:08.348507 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:08.348465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwhf7" event={"ID":"8be26753-c6b2-4a1f-a7e9-9beb72381360","Type":"ContainerStarted","Data":"648ec522ef2aa68f67687f6960fda8e547e377acfe2c0d77e154531a4cb76cc3"}
Sep 10 11:53:09.353268 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:09.353177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwhf7" event={"ID":"8be26753-c6b2-4a1f-a7e9-9beb72381360","Type":"ContainerStarted","Data":"d5cfe781fb839ac8312f942dfbfc9b9f575cb9c00dd36ca8b44d7bd2d12bdcd2"}
Sep 10 11:53:09.375080 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:09.375033    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nwhf7" podStartSLOduration=17.17099622 podStartE2EDuration="19.375020957s" totalImagesPullingTime="2.204024737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 11:52:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:09.373154247 +0000 UTC m=+90.953081633" watchObservedRunningTime="2026-09-10 11:53:09.375020957 +0000 UTC m=+90.954948380"
Sep 10 11:53:09.941255 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:09.941223    2573 scope.go:117] "RemoveContainer" containerID="8ff47d85f426052968f3901a39fdc0a5306f7f2769b9770cc2d6778ac67c67c5"
Sep 10 11:53:10.357994 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:10.357914    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 11:53:10.358346 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:10.358041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" event={"ID":"7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc","Type":"ContainerStarted","Data":"5cbd9659d075b5e7f0f60dab4692a2d6030e44cfb9ec43bd635db2795d62a5cd"}
Sep 10 11:53:10.358513 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:10.358494    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:53:10.382722 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:10.382677    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-hwhk6" podStartSLOduration=20.320496047 podStartE2EDuration="25.382664052s" totalImagesPullingTime="5.062168005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:10.380655475 +0000 UTC m=+91.960582873" watchObservedRunningTime="2026-09-10 11:53:10.382664052 +0000 UTC m=+91.962591438"
Sep 10 11:53:10.567178 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:10.567151    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-hwhk6"
Sep 10 11:53:17.683165 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.683130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:17.683551 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.683186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:53:17.683869 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.683848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cb09d4e-faf7-49d9-81e6-1d7504d65425-service-ca-bundle\") pod \"router-default-54478f5d74-kjsrv\" (UID: \"0cb09d4e-faf7-49d9-81e6-1d7504d65425\") " pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:17.685487 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.685472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:17.685614 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.685599    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a486eacf-3b31-4de8-9a5d-fcbff00af988-metrics-tls\") pod \"dns-default-cgs5d\" (UID: \"a486eacf-3b31-4de8-9a5d-fcbff00af988\") " pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:53:17.963590 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.963519    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q76h9\""
Sep 10 11:53:17.966120 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.966087    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54478f5d74-kjsrv"]
Sep 10 11:53:17.969777 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:17.969752    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb09d4e_faf7_49d9_81e6_1d7504d65425.slice/crio-7c7563f484a9545236c63a5b44927a656b7eece707127c602fb14b8ba433395f WatchSource:0}: Error finding container 7c7563f484a9545236c63a5b44927a656b7eece707127c602fb14b8ba433395f: Status 404 returned error can't find the container with id 7c7563f484a9545236c63a5b44927a656b7eece707127c602fb14b8ba433395f
Sep 10 11:53:17.970616 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.970595    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:53:17.971032 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.971015    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"]
Sep 10 11:53:17.974751 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.974704    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:17.977223 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.977187    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 11:53:17.977723 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.977704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 11:53:17.977835 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.977709    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 11:53:17.977835 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.977813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 11:53:17.988359 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:17.988318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"]
Sep 10 11:53:18.086053 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.086013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw76\" (UniqueName: \"kubernetes.io/projected/53afe210-3c06-40bb-b346-273c2c62f938-kube-api-access-pjw76\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.086209 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.086113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53afe210-3c06-40bb-b346-273c2c62f938-tmp\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.086209 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.086156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/53afe210-3c06-40bb-b346-273c2c62f938-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.187426 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.187392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53afe210-3c06-40bb-b346-273c2c62f938-tmp\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.187426 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.187432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/53afe210-3c06-40bb-b346-273c2c62f938-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.187656 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.187458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw76\" (UniqueName: \"kubernetes.io/projected/53afe210-3c06-40bb-b346-273c2c62f938-kube-api-access-pjw76\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.187802 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.187780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53afe210-3c06-40bb-b346-273c2c62f938-tmp\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.190336 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.190286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/53afe210-3c06-40bb-b346-273c2c62f938-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.200646 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.200622    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw76\" (UniqueName: \"kubernetes.io/projected/53afe210-3c06-40bb-b346-273c2c62f938-kube-api-access-pjw76\") pod \"klusterlet-addon-workmgr-5dcc655dcc-qvdcf\" (UID: \"53afe210-3c06-40bb-b346-273c2c62f938\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.261864 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.261842    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cgs5d"]
Sep 10 11:53:18.263980 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:18.263956    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda486eacf_3b31_4de8_9a5d_fcbff00af988.slice/crio-bdab8c2db55bb25b6a24432860a5b6e09c0aed8607fe5850876ffb220d453733 WatchSource:0}: Error finding container bdab8c2db55bb25b6a24432860a5b6e09c0aed8607fe5850876ffb220d453733: Status 404 returned error can't find the container with id bdab8c2db55bb25b6a24432860a5b6e09c0aed8607fe5850876ffb220d453733
Sep 10 11:53:18.292889 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.292865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:18.383986 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.383951    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgs5d" event={"ID":"a486eacf-3b31-4de8-9a5d-fcbff00af988","Type":"ContainerStarted","Data":"bdab8c2db55bb25b6a24432860a5b6e09c0aed8607fe5850876ffb220d453733"}
Sep 10 11:53:18.385629 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.385606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54478f5d74-kjsrv" event={"ID":"0cb09d4e-faf7-49d9-81e6-1d7504d65425","Type":"ContainerStarted","Data":"b2e0a023237eeb85128fd2714c964dfd9a342c92b27e7bc7f1805f58996f50d1"}
Sep 10 11:53:18.385733 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.385648    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54478f5d74-kjsrv" event={"ID":"0cb09d4e-faf7-49d9-81e6-1d7504d65425","Type":"ContainerStarted","Data":"7c7563f484a9545236c63a5b44927a656b7eece707127c602fb14b8ba433395f"}
Sep 10 11:53:18.569422 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.569290    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-54478f5d74-kjsrv" podStartSLOduration=33.56927309 podStartE2EDuration="33.56927309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:18.412456532 +0000 UTC m=+99.992383920" watchObservedRunningTime="2026-09-10 11:53:18.56927309 +0000 UTC m=+100.149200488"
Sep 10 11:53:18.569856 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.569838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"]
Sep 10 11:53:18.572888 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:18.572869    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53afe210_3c06_40bb_b346_273c2c62f938.slice/crio-21658c4cef800405eb25643d7900a93a8bd1fdb2ecbf2592d917dea6e8131d5a WatchSource:0}: Error finding container 21658c4cef800405eb25643d7900a93a8bd1fdb2ecbf2592d917dea6e8131d5a: Status 404 returned error can't find the container with id 21658c4cef800405eb25643d7900a93a8bd1fdb2ecbf2592d917dea6e8131d5a
Sep 10 11:53:18.686287 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.686256    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:18.688820 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.688795    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:18.876984 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.876900    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:18.880050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.880022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.882669 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882497    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 11:53:18.882669 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882550    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 11:53:18.882985 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882766    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 11:53:18.882985 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882772    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lrm5q\""
Sep 10 11:53:18.882985 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882809    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 11:53:18.882985 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.882911    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 11:53:18.883207 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.883089    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 11:53:18.883592 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.883566    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 11:53:18.894335 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.894313    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:18.993234 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.993422 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.993422 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnc4\" (UniqueName: \"kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.993422 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.993422 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993382    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:18.993597 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:18.993432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.093833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.093797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094000 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.093865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094000 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.093914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnc4\" (UniqueName: \"kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094000 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.093950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094165 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.094129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094228 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.094171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094584 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.094547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.094878 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.094789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.095355 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.095291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.096717 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.096691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.096894 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.096874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.111339 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.111262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnc4\" (UniqueName: \"kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4\") pod \"console-dcc8584b5-gb5dt\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") " pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.191218 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.191190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:19.394123 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.392741    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf" event={"ID":"53afe210-3c06-40bb-b346-273c2c62f938","Type":"ContainerStarted","Data":"21658c4cef800405eb25643d7900a93a8bd1fdb2ecbf2592d917dea6e8131d5a"}
Sep 10 11:53:19.394123 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.392824    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:19.394741 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.394723    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54478f5d74-kjsrv"
Sep 10 11:53:19.497606 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:19.497482    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:20.398843 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:20.398757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc8584b5-gb5dt" event={"ID":"2466a590-c492-418e-b5c1-2bdc2dc49b1a","Type":"ContainerStarted","Data":"dffbb9ba08725b9224d579d03fb3a8fdeba4c8c94cd9ec8003b387d2b39ba0f5"}
Sep 10 11:53:23.292842 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:23.292812    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-77jx8"
Sep 10 11:53:25.425188 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.425149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf" event={"ID":"53afe210-3c06-40bb-b346-273c2c62f938","Type":"ContainerStarted","Data":"01a0d59a306fda21336c7544ae07f601b2c1454db37e53f73b44e178aefcb513"}
Sep 10 11:53:25.425665 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.425353    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:25.427164 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.427138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgs5d" event={"ID":"a486eacf-3b31-4de8-9a5d-fcbff00af988","Type":"ContainerStarted","Data":"f6ad15106b862d1d19c5179a72150b5dbd9a1f5f6beca32673ae476783c059a4"}
Sep 10 11:53:25.427287 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.427188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cgs5d" event={"ID":"a486eacf-3b31-4de8-9a5d-fcbff00af988","Type":"ContainerStarted","Data":"2615d302134bf4b52a8fb10b732d7da6b3c39408c29461cc81f67eec32728220"}
Sep 10 11:53:25.427287 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.427217    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:53:25.427426 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.427406    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf"
Sep 10 11:53:25.428773 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.428751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc8584b5-gb5dt" event={"ID":"2466a590-c492-418e-b5c1-2bdc2dc49b1a","Type":"ContainerStarted","Data":"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"}
Sep 10 11:53:25.444681 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.444638    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dcc655dcc-qvdcf" podStartSLOduration=2.681080339 podStartE2EDuration="8.444624324s" totalImagesPullingTime="5.763543985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:25.442622668 +0000 UTC m=+107.022550065" watchObservedRunningTime="2026-09-10 11:53:25.444624324 +0000 UTC m=+107.024551709"
Sep 10 11:53:25.483249 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.483206    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cgs5d" podStartSLOduration=66.44061308 podStartE2EDuration="1m12.483192686s" totalImagesPullingTime="6.042579606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:25.48269252 +0000 UTC m=+107.062619907" watchObservedRunningTime="2026-09-10 11:53:25.483192686 +0000 UTC m=+107.063120069"
Sep 10 11:53:25.502569 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:25.502530    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcc8584b5-gb5dt" podStartSLOduration=3.3244354720000002 podStartE2EDuration="7.502518538s" totalImagesPullingTime="4.178083066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:25.501676755 +0000 UTC m=+107.081604143" watchObservedRunningTime="2026-09-10 11:53:25.502518538 +0000 UTC m=+107.082445924"
Sep 10 11:53:29.191541 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.191508    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:29.191934 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.191559    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:29.195958 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.195940    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:29.446771 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.446696    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:29.820476 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.820397    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"]
Sep 10 11:53:29.825039 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.825010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:29.835448 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.835430    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 11:53:29.836227 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.836209    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 11:53:29.836339 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.836248    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 11:53:29.836339 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.836259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 11:53:29.837071 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.837056    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pd7pv\""
Sep 10 11:53:29.837275 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.837263    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 11:53:29.845369 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.845350    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"]
Sep 10 11:53:29.866454 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.866428    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"]
Sep 10 11:53:29.870036 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.870018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.873545 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.873521    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 11:53:29.874729 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.874709    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 11:53:29.875342 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.875297    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dhv2w\""
Sep 10 11:53:29.892466 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.892440    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 11:53:29.925143 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.925113    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mx694"]
Sep 10 11:53:29.929744 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.929724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:29.934518 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.934490    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 11:53:29.934630 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.934522    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 11:53:29.934630 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.934496    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kl6rx\""
Sep 10 11:53:29.938629 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.938609    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 11:53:29.989777 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989731    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.989777 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64b45048-9461-44fa-b098-e6026ebd912d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.989777 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989784    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.990042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.990042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989882    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:29.990042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:29.990042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.989998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4b857-a978-4980-bd89-5868b5b0dff8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:29.990042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.990021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxgc\" (UniqueName: \"kubernetes.io/projected/48f4b857-a978-4980-bd89-5868b5b0dff8-kube-api-access-tjxgc\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:29.990240 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.990054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkhv\" (UniqueName: \"kubernetes.io/projected/64b45048-9461-44fa-b098-e6026ebd912d-kube-api-access-llkhv\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:29.990240 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:29.990074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.004572 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.004537    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"]
Sep 10 11:53:30.091510 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-sys\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091510 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.091728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-textfile\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4b857-a978-4980-bd89-5868b5b0dff8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.091728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091656    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-root\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091728 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxgc\" (UniqueName: \"kubernetes.io/projected/48f4b857-a978-4980-bd89-5868b5b0dff8-kube-api-access-tjxgc\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-tls\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkhv\" (UniqueName: \"kubernetes.io/projected/64b45048-9461-44fa-b098-e6026ebd912d-kube-api-access-llkhv\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-wtmp\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.091966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64b45048-9461-44fa-b098-e6026ebd912d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.091975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfbx\" (UniqueName: \"kubernetes.io/projected/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-kube-api-access-7wfbx\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.092261 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-metrics-client-ca\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.092583 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f4b857-a978-4980-bd89-5868b5b0dff8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.092779 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.092752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/64b45048-9461-44fa-b098-e6026ebd912d-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.093415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.093390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.093534 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.093512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.095021 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.094990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.095254 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.095233    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/48f4b857-a978-4980-bd89-5868b5b0dff8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.095327 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.095281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.095648 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.095629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64b45048-9461-44fa-b098-e6026ebd912d-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.103086 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.103056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxgc\" (UniqueName: \"kubernetes.io/projected/48f4b857-a978-4980-bd89-5868b5b0dff8-kube-api-access-tjxgc\") pod \"openshift-state-metrics-657fd6c54-gwvrx\" (UID: \"48f4b857-a978-4980-bd89-5868b5b0dff8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.103086 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.103077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkhv\" (UniqueName: \"kubernetes.io/projected/64b45048-9461-44fa-b098-e6026ebd912d-kube-api-access-llkhv\") pod \"kube-state-metrics-74cb57df7-8vjbb\" (UID: \"64b45048-9461-44fa-b098-e6026ebd912d\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.137186 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.134025    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"
Sep 10 11:53:30.181470 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.181434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.193219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-wtmp\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.193579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-wtmp\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.193292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfbx\" (UniqueName: \"kubernetes.io/projected/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-kube-api-access-7wfbx\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194485    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-metrics-client-ca\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-sys\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-textfile\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-root\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.195100 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.194693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-tls\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.197888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.196588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-sys\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.197888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.197376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-textfile\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.197888 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.197840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-root\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.198862 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.198504    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-metrics-client-ca\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.203526 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.203458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-tls\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.204091 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.204054    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.222854 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.222792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfbx\" (UniqueName: \"kubernetes.io/projected/d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad-kube-api-access-7wfbx\") pod \"node-exporter-mx694\" (UID: \"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad\") " pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.241547 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.241476    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx694"
Sep 10 11:53:30.412335 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.412285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx"]
Sep 10 11:53:30.414636 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:30.414608    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f4b857_a978_4980_bd89_5868b5b0dff8.slice/crio-218a922ba005ab682033b30894f3d379fb3fd30934c7c2efc3a12137a1b20ecb WatchSource:0}: Error finding container 218a922ba005ab682033b30894f3d379fb3fd30934c7c2efc3a12137a1b20ecb: Status 404 returned error can't find the container with id 218a922ba005ab682033b30894f3d379fb3fd30934c7c2efc3a12137a1b20ecb
Sep 10 11:53:30.446469 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.446439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx694" event={"ID":"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad","Type":"ContainerStarted","Data":"03a8d6bbfbe7b4e4d15a8e736a1eeb82635c96cc5c54146d8219de885d595cda"}
Sep 10 11:53:30.447889 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.447854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx" event={"ID":"48f4b857-a978-4980-bd89-5868b5b0dff8","Type":"ContainerStarted","Data":"218a922ba005ab682033b30894f3d379fb3fd30934c7c2efc3a12137a1b20ecb"}
Sep 10 11:53:30.469385 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.469361    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb"]
Sep 10 11:53:30.472090 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:30.472061    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b45048_9461_44fa_b098_e6026ebd912d.slice/crio-79ef16d078c70520cadda05d06ce76ef9c64f830768bee53540abc2b29c7a9fe WatchSource:0}: Error finding container 79ef16d078c70520cadda05d06ce76ef9c64f830768bee53540abc2b29c7a9fe: Status 404 returned error can't find the container with id 79ef16d078c70520cadda05d06ce76ef9c64f830768bee53540abc2b29c7a9fe
Sep 10 11:53:30.931471 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.931440    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 11:53:30.933848 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.933829    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:30.939101 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939079    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 11:53:30.939232 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939101    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 11:53:30.939232 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939126    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 11:53:30.939232 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939081    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 11:53:30.939417 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939348    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 11:53:30.939417 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939387    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 11:53:30.939644 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939627    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 11:53:30.939743 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.939727    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 11:53:30.940238 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.940220    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9k8vs\""
Sep 10 11:53:30.950358 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.950333    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 11:53:30.958219 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:30.958194    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 11:53:31.101711 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.101858 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.101858 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101742    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.101858 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-web-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.101858 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.101858 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99rq\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-kube-api-access-r99rq\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101932    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-out\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102050 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.101985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102218 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.102045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.102218 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.102078    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203096 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-web-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203096 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203127    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99rq\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-kube-api-access-r99rq\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203161    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-out\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.203608 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.203493    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.205075 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.205012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.205496 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.205471    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.207959 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.207930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-volume\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.210586 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.210178    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.210586 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.210510    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212177 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.211268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-web-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212177 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.211582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-config-out\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212177 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.212004    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212407 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.212251    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212407 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.212400    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.212953 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.212921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.214493 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.214450    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.217945 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.217903    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99rq\" (UniqueName: \"kubernetes.io/projected/b7316fb9-7bd9-47f3-b4ac-c26f12c01f65-kube-api-access-r99rq\") pod \"alertmanager-main-0\" (UID: \"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.244257 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.244047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 11:53:31.455206 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.455051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb" event={"ID":"64b45048-9461-44fa-b098-e6026ebd912d","Type":"ContainerStarted","Data":"79ef16d078c70520cadda05d06ce76ef9c64f830768bee53540abc2b29c7a9fe"}
Sep 10 11:53:31.458364 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.458269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx" event={"ID":"48f4b857-a978-4980-bd89-5868b5b0dff8","Type":"ContainerStarted","Data":"ef13c17a9eab34ab2d504fd6d3525aa5ef1da88a7308032f2fda43dccb184be4"}
Sep 10 11:53:31.458469 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.458389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx" event={"ID":"48f4b857-a978-4980-bd89-5868b5b0dff8","Type":"ContainerStarted","Data":"af12af3012f3b4b948c1179dddb89d262ca80e55aba9d3a9b1f2f0b093169215"}
Sep 10 11:53:31.670415 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:31.670376    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 11:53:32.007896 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:32.007861    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7316fb9_7bd9_47f3_b4ac_c26f12c01f65.slice/crio-476f20ee2017e778b196bf0c66e70b20c4bdb695ad46e2cf78dc3d7fe71b198d WatchSource:0}: Error finding container 476f20ee2017e778b196bf0c66e70b20c4bdb695ad46e2cf78dc3d7fe71b198d: Status 404 returned error can't find the container with id 476f20ee2017e778b196bf0c66e70b20c4bdb695ad46e2cf78dc3d7fe71b198d
Sep 10 11:53:32.165908 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.165875    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:32.463364 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.463328    2573 generic.go:356] "Generic (PLEG): container finished" podID="d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad" containerID="acd39bbc163d5b61203ae36a7b268d17b7aa65df58581ae07c229960377eb359" exitCode=0
Sep 10 11:53:32.463825 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.463407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx694" event={"ID":"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad","Type":"ContainerDied","Data":"acd39bbc163d5b61203ae36a7b268d17b7aa65df58581ae07c229960377eb359"}
Sep 10 11:53:32.465956 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.465930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb" event={"ID":"64b45048-9461-44fa-b098-e6026ebd912d","Type":"ContainerStarted","Data":"53e10d0a5ec565e5d98e4318a8d3ac501f2f62ee001fad591244382058850d59"}
Sep 10 11:53:32.466072 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.465985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb" event={"ID":"64b45048-9461-44fa-b098-e6026ebd912d","Type":"ContainerStarted","Data":"56332eb20d71a54e46df921eadda1a937d9d14c557be8bf433755692cef09475"}
Sep 10 11:53:32.466072 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.466012    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb" event={"ID":"64b45048-9461-44fa-b098-e6026ebd912d","Type":"ContainerStarted","Data":"baeabd3287071a4b55fdeac066ba5ef0eb5cf2e1616e911ab6d241038ac827e1"}
Sep 10 11:53:32.468273 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.468251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx" event={"ID":"48f4b857-a978-4980-bd89-5868b5b0dff8","Type":"ContainerStarted","Data":"d2971d8c1eba87b100b73242dfe653e003bd852cd4e3254f3b2f7aee105ebeb8"}
Sep 10 11:53:32.469727 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.469702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"476f20ee2017e778b196bf0c66e70b20c4bdb695ad46e2cf78dc3d7fe71b198d"}
Sep 10 11:53:32.511280 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.509998    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-gwvrx" podStartSLOduration=1.993354546 podStartE2EDuration="3.509978897s" totalImagesPullingTime="1.516624351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:32.509600611 +0000 UTC m=+114.089527998" watchObservedRunningTime="2026-09-10 11:53:32.509978897 +0000 UTC m=+114.089906285"
Sep 10 11:53:32.533159 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.532743    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-8vjbb" podStartSLOduration=1.946743987 podStartE2EDuration="3.532722041s" totalImagesPullingTime="1.585978054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:32.531437298 +0000 UTC m=+114.111364685" watchObservedRunningTime="2026-09-10 11:53:32.532722041 +0000 UTC m=+114.112649429"
Sep 10 11:53:32.766316 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.766268    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d9794884-c55p7"]
Sep 10 11:53:32.770063 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.770047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.773645 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.773578    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s4jz8\""
Sep 10 11:53:32.773877 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.773862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 11:53:32.773966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.773906    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 11:53:32.773966 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.773958    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4fdf3raog5dga\""
Sep 10 11:53:32.774295 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.774260    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 11:53:32.774295 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.774268    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 11:53:32.774477 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.774346    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 11:53:32.809634 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.809606    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d9794884-c55p7"]
Sep 10 11:53:32.923239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gbt\" (UniqueName: \"kubernetes.io/projected/4af3e680-8418-4c18-9781-c249db2c0ff3-kube-api-access-r6gbt\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923239 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923251    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923485 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923272    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923485 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923485 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af3e680-8418-4c18-9781-c249db2c0ff3-metrics-client-ca\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923587 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923587 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:32.923587 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:32.923533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-grpc-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.024747 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gbt\" (UniqueName: \"kubernetes.io/projected/4af3e680-8418-4c18-9781-c249db2c0ff3-kube-api-access-r6gbt\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.024747 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.024747 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af3e680-8418-4c18-9781-c249db2c0ff3-metrics-client-ca\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.024924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-grpc-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.025700 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.025672    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af3e680-8418-4c18-9781-c249db2c0ff3-metrics-client-ca\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.027641 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.027614    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.027867 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.027844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.027920 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.027888    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.027976 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.027954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.028025 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.028010    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.028226 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.028210    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af3e680-8418-4c18-9781-c249db2c0ff3-secret-grpc-tls\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.033560 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.033541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gbt\" (UniqueName: \"kubernetes.io/projected/4af3e680-8418-4c18-9781-c249db2c0ff3-kube-api-access-r6gbt\") pod \"thanos-querier-d9794884-c55p7\" (UID: \"4af3e680-8418-4c18-9781-c249db2c0ff3\") " pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.079647 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.079616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:33.359001 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.358977    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d9794884-c55p7"]
Sep 10 11:53:33.361423 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:33.361385    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af3e680_8418_4c18_9781_c249db2c0ff3.slice/crio-bf08cfd69e63bdaa492ec55b60ff35291ab5c2476fcac92cdc86e7d3d489225a WatchSource:0}: Error finding container bf08cfd69e63bdaa492ec55b60ff35291ab5c2476fcac92cdc86e7d3d489225a: Status 404 returned error can't find the container with id bf08cfd69e63bdaa492ec55b60ff35291ab5c2476fcac92cdc86e7d3d489225a
Sep 10 11:53:33.475627 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.475590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx694" event={"ID":"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad","Type":"ContainerStarted","Data":"d1cb889390958fc549838d0287ee36e9864a5b94108a3ec52c052b83a4c8b354"}
Sep 10 11:53:33.476011 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.475651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx694" event={"ID":"d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad","Type":"ContainerStarted","Data":"4f562f0c67b7a7c360088c91e87d9e8493bc415e531fdc4421d311bc2007f2f2"}
Sep 10 11:53:33.476954 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.476927    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"bf08cfd69e63bdaa492ec55b60ff35291ab5c2476fcac92cdc86e7d3d489225a"}
Sep 10 11:53:33.514896 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:33.514848    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mx694" podStartSLOduration=3.280085716 podStartE2EDuration="4.51483173s" totalImagesPullingTime="1.144551029s" totalInitContainerRuntime="90.194985ms" isStatefulPod=true podCreationTimestamp="2026-09-10 11:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:33.513199134 +0000 UTC m=+115.093126533" watchObservedRunningTime="2026-09-10 11:53:33.51483173 +0000 UTC m=+115.094759116"
Sep 10 11:53:35.433735 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.433702    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cgs5d"
Sep 10 11:53:35.489098 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.489065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"cf0b8d4ea17ac581c95ea8954f2d8810f977b0dda777463772d2986ebfaacb06"}
Sep 10 11:53:35.489253 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.489138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"5479825e2b8d4116a8ff039afdfd2524dfa0ac1a4cdcf6539b4ba41e43afb223"}
Sep 10 11:53:35.489253 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.489164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"d7becfeed0c5ddc1296daf50ec5d72a49861fb2307b0cb75d29333bf776e2e5f"}
Sep 10 11:53:35.987340 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.987292    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 11:53:35.991964 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:35.991930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.001518 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.001493    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 11:53:36.002919 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.002888    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tmkvf\""
Sep 10 11:53:36.003026 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.002929    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 11:53:36.003026 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.002987    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 11:53:36.003026 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003016    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 11:53:36.003243 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003045    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 11:53:36.003243 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003086    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 11:53:36.003243 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003180    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 11:53:36.003401 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003273    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 11:53:36.003401 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003369    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 11:53:36.003401 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003375    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 11:53:36.003401 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003394    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 11:53:36.003565 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003526    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 11:53:36.003604 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003584    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 11:53:36.003924 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003824    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-88ubqaa9covsg\""
Sep 10 11:53:36.003924 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.003879    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 11:53:36.005432 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.005414    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 11:53:36.013010 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.012988    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 11:53:36.059493 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059493 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059622    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.059716 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq75\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-kube-api-access-chq75\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.059949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.060016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.060046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.060063    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.060114 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.060084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160479 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160643 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160719 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160777 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160834 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160898 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160831    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160898 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160999 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160999 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.160999 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.160969    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161144 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161144 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161144 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161144 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161144 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161585 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161585 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161585 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.161585 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.161546    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.162172 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.162139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.162172 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.162164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.162828 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.162526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.164569 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.164548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.164732 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.164712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq75\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-kube-api-access-chq75\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.164880 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.164862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.165641 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.165616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.165731 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.165704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166014 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.165992    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166210 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166336 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166545 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166523    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166632 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166801 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.166928 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.166905    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.167201 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.167173    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.167373 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.167354    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.167453 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.167426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.167510 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.167490    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.168520 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.168502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-config\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.176500 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.176477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq75\" (UniqueName: \"kubernetes.io/projected/1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b-kube-api-access-chq75\") pod \"prometheus-k8s-0\" (UID: \"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.302702 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.302679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:36.496149 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.496121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"c25bcd6b5b57e615726074319c6589adfc5f87e948b0513503126a3e2e4441ab"}
Sep 10 11:53:36.496532 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.496239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"a850c4344a11fd61851dfacd0fc1a1edaaed436c504e72230f29539d33c42271"}
Sep 10 11:53:36.498034 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.498003    2573 generic.go:356] "Generic (PLEG): container finished" podID="b7316fb9-7bd9-47f3-b4ac-c26f12c01f65" containerID="ced89572b93bc4ed69f4190cddcc69f4bf36e3f3ca6445582e353596e2638f35" exitCode=0
Sep 10 11:53:36.498133 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.498083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerDied","Data":"ced89572b93bc4ed69f4190cddcc69f4bf36e3f3ca6445582e353596e2638f35"}
Sep 10 11:53:36.599628 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:36.599552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 11:53:36.605127 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:36.605097    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1869aad1_aaaf_492e_a4ad_fb0bb9f4a06b.slice/crio-a085a0f7a300416cd36c241bfe09508dd2f536156615d2d3dbdeb12f13918a5e WatchSource:0}: Error finding container a085a0f7a300416cd36c241bfe09508dd2f536156615d2d3dbdeb12f13918a5e: Status 404 returned error can't find the container with id a085a0f7a300416cd36c241bfe09508dd2f536156615d2d3dbdeb12f13918a5e
Sep 10 11:53:37.508166 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.508121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" event={"ID":"4af3e680-8418-4c18-9781-c249db2c0ff3","Type":"ContainerStarted","Data":"59275880168c500d8626108477b37724307394acaaebf5f9fd9de7e8eb34d0cc"}
Sep 10 11:53:37.508609 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.508290    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:37.509739 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.509717    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b" containerID="dec4def2bdce13a06f06ea03db25fcd58cbe14fbb5ba53659cf957841283bf3e" exitCode=0
Sep 10 11:53:37.509823 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.509763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerDied","Data":"dec4def2bdce13a06f06ea03db25fcd58cbe14fbb5ba53659cf957841283bf3e"}
Sep 10 11:53:37.509823 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.509807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"a085a0f7a300416cd36c241bfe09508dd2f536156615d2d3dbdeb12f13918a5e"}
Sep 10 11:53:37.536042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:37.535978    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d9794884-c55p7" podStartSLOduration=2.8016330910000002 podStartE2EDuration="5.535963687s" totalImagesPullingTime="2.734330596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:37.534597395 +0000 UTC m=+119.114524806" watchObservedRunningTime="2026-09-10 11:53:37.535963687 +0000 UTC m=+119.115891103"
Sep 10 11:53:38.518615 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"556c6c940df06c142ae7c5926a92c90d6d2e6bb2de363e22e927964ddaae3884"}
Sep 10 11:53:38.519054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"8e79ae6a7d4e1aa435eb4b333becd97a65d2f82a746433dbd0a6dde5a9a69ab7"}
Sep 10 11:53:38.519054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"ac8fc0bcde34de3e99bf8aa9c9971eb6db19dae93065c3d54cc1e3a95bfa07f5"}
Sep 10 11:53:38.519054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"8454b350b4ea2e5dc4af08dcf06a3513db230d4aab93d78c6c9cb304f02c9df8"}
Sep 10 11:53:38.519054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"809836fbaf0b1b5380f3d18f59b3bffcc20df92bc3fbe96fba59ffebb891525c"}
Sep 10 11:53:38.519054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.518820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b7316fb9-7bd9-47f3-b4ac-c26f12c01f65","Type":"ContainerStarted","Data":"d5a07cf765970c251e0e0f53510609823ebe5fcafdb99ec2930a841ecc6f6770"}
Sep 10 11:53:38.551172 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:38.551115    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.892018702 podStartE2EDuration="8.551100313s" totalImagesPullingTime="5.637773417s" totalInitContainerRuntime="21.308194ms" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:38.549437921 +0000 UTC m=+120.129365315" watchObservedRunningTime="2026-09-10 11:53:38.551100313 +0000 UTC m=+120.131027699"
Sep 10 11:53:41.540588 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"989ec4655631914e93526e3cd6beb9dab4c4a848d022795c7bc505de73e9f63f"}
Sep 10 11:53:41.540962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540619    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"fd535f16f68ce46785acf3ae1715a95724fa3be03674719fb5d6820cce49149f"}
Sep 10 11:53:41.540962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540647    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"2287a40a6ed3cf33f6896a773162f0dd454db7613f9e482f224846e1ac20485e"}
Sep 10 11:53:41.540962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"4227dbee1535141d39b922f6fc647badd93c88f3117245fe85b7401cf0fbaaf2"}
Sep 10 11:53:41.540962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"7a3f932e8b6ac20ca0f753464cf1760a6da67084cd655f52ff0f1f826d0c1e2d"}
Sep 10 11:53:41.540962 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.540722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b","Type":"ContainerStarted","Data":"c22fc8d966267f0aa7c361f618af425613d08d55c792ad700ec62f026a6a1fc8"}
Sep 10 11:53:41.595541 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:41.595481    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.503743766 podStartE2EDuration="6.595462417s" totalImagesPullingTime="3.073873603s" totalInitContainerRuntime="17.845048ms" isStatefulPod=false podCreationTimestamp="2026-09-10 11:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:41.592805853 +0000 UTC m=+123.172733241" watchObservedRunningTime="2026-09-10 11:53:41.595462417 +0000 UTC m=+123.175389804"
Sep 10 11:53:43.524743 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:43.524712    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d9794884-c55p7"
Sep 10 11:53:46.303573 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:46.303534    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:53:48.785800 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:48.785762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:53:48.788318 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:48.788268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad6550a-c538-4e96-9e17-5019310b5341-metrics-certs\") pod \"network-metrics-daemon-4brxb\" (UID: \"fad6550a-c538-4e96-9e17-5019310b5341\") " pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:53:49.052335 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:49.052236    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zrvwr\""
Sep 10 11:53:49.060498 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:49.060479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4brxb"
Sep 10 11:53:49.336061 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:49.336030    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4brxb"]
Sep 10 11:53:49.338644 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:53:49.338621    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad6550a_c538_4e96_9e17_5019310b5341.slice/crio-3f600d5d880b0827757dd2d0a5af92fb2a0b22a27302925562a38cc4da9b63c3 WatchSource:0}: Error finding container 3f600d5d880b0827757dd2d0a5af92fb2a0b22a27302925562a38cc4da9b63c3: Status 404 returned error can't find the container with id 3f600d5d880b0827757dd2d0a5af92fb2a0b22a27302925562a38cc4da9b63c3
Sep 10 11:53:49.577606 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:49.577569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4brxb" event={"ID":"fad6550a-c538-4e96-9e17-5019310b5341","Type":"ContainerStarted","Data":"3f600d5d880b0827757dd2d0a5af92fb2a0b22a27302925562a38cc4da9b63c3"}
Sep 10 11:53:50.584204 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:50.584116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4brxb" event={"ID":"fad6550a-c538-4e96-9e17-5019310b5341","Type":"ContainerStarted","Data":"237aa28859d0b882b7379a6aee737af5d310d6a3ff3fd6d186e23044d0420573"}
Sep 10 11:53:50.584204 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:50.584183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4brxb" event={"ID":"fad6550a-c538-4e96-9e17-5019310b5341","Type":"ContainerStarted","Data":"52dbc61a26c14c01d955628b5182677b3810d333aab8f952093386545636c94d"}
Sep 10 11:53:50.608978 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:50.608925    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4brxb" podStartSLOduration=130.758895863 podStartE2EDuration="2m11.60891033s" totalImagesPullingTime="850.014467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:53:50.607475707 +0000 UTC m=+132.187403094" watchObservedRunningTime="2026-09-10 11:53:50.60891033 +0000 UTC m=+132.188837715"
Sep 10 11:53:57.191610 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.191568    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dcc8584b5-gb5dt" podUID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" containerName="console" containerID="cri-o://2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48" gracePeriod=15
Sep 10 11:53:57.438319 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.438283    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:57.564464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564381    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564414    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564464 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vnc4\" (UniqueName: \"kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564734 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564476    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564734 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564498    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564734 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564550    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config\") pod \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\" (UID: \"2466a590-c492-418e-b5c1-2bdc2dc49b1a\") "
Sep 10 11:53:57.564893 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564831    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config" (OuterVolumeSpecName: "console-config") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:53:57.564956 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564927    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:53:57.565022 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.564935    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 11:53:57.566772 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.566744    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:53:57.566960 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.566931    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4" (OuterVolumeSpecName: "kube-api-access-8vnc4") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "kube-api-access-8vnc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 11:53:57.567026 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.566965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2466a590-c492-418e-b5c1-2bdc2dc49b1a" (UID: "2466a590-c492-418e-b5c1-2bdc2dc49b1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 11:53:57.613371 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.613341    2573 generic.go:356] "Generic (PLEG): container finished" podID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" containerID="2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48" exitCode=0
Sep 10 11:53:57.613471 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.613387    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcc8584b5-gb5dt"
Sep 10 11:53:57.613471 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.613420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc8584b5-gb5dt" event={"ID":"2466a590-c492-418e-b5c1-2bdc2dc49b1a","Type":"ContainerDied","Data":"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"}
Sep 10 11:53:57.613538 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.613474    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcc8584b5-gb5dt" event={"ID":"2466a590-c492-418e-b5c1-2bdc2dc49b1a","Type":"ContainerDied","Data":"dffbb9ba08725b9224d579d03fb3a8fdeba4c8c94cd9ec8003b387d2b39ba0f5"}
Sep 10 11:53:57.613538 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.613495    2573 scope.go:117] "RemoveContainer" containerID="2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"
Sep 10 11:53:57.624822 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.624806    2573 scope.go:117] "RemoveContainer" containerID="2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"
Sep 10 11:53:57.625162 ip-10-0-134-58 kubenswrapper[2573]: E0910 11:53:57.625129    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48\": container with ID starting with 2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48 not found: ID does not exist" containerID="2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"
Sep 10 11:53:57.625250 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.625167    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48"} err="failed to get container status \"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48\": rpc error: code = NotFound desc = could not find container \"2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48\": container with ID starting with 2cef58237123ba06d38b21aab0678321077c9d4b901cf197a74dc2e0f2fccb48 not found: ID does not exist"
Sep 10 11:53:57.638580 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.638556    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:57.642961 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.642941    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcc8584b5-gb5dt"]
Sep 10 11:53:57.666060 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666040    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:57.666060 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666061    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-oauth-serving-cert\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:57.666159 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666070    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-oauth-config\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:57.666159 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666079    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-config\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:57.666159 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666088    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2466a590-c492-418e-b5c1-2bdc2dc49b1a-console-serving-cert\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:57.666159 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:57.666095    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vnc4\" (UniqueName: \"kubernetes.io/projected/2466a590-c492-418e-b5c1-2bdc2dc49b1a-kube-api-access-8vnc4\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 11:53:58.946213 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:53:58.946181    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" path="/var/lib/kubelet/pods/2466a590-c492-418e-b5c1-2bdc2dc49b1a/volumes"
Sep 10 11:54:17.703567 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:17.703528    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2c142af-6a36-4984-ba8f-8255afadd1e4" containerID="4941b563f78314a56539f0a06504f3fa04fe845574f43aa84c03f22b209875ca" exitCode=0
Sep 10 11:54:17.703935 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:17.703596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2" event={"ID":"a2c142af-6a36-4984-ba8f-8255afadd1e4","Type":"ContainerDied","Data":"4941b563f78314a56539f0a06504f3fa04fe845574f43aa84c03f22b209875ca"}
Sep 10 11:54:17.703982 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:17.703944    2573 scope.go:117] "RemoveContainer" containerID="4941b563f78314a56539f0a06504f3fa04fe845574f43aa84c03f22b209875ca"
Sep 10 11:54:18.709623 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:18.709588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-hr2b2" event={"ID":"a2c142af-6a36-4984-ba8f-8255afadd1e4","Type":"ContainerStarted","Data":"ccf4cdc5951c73a910b7ba5d718945f966dc8a5468c08cee6ab0da259f52917b"}
Sep 10 11:54:36.307634 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:36.303448    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:54:36.323574 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:36.323549    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:54:36.811866 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:54:36.811841    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 11:56:38.855066 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:56:38.855041    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 11:56:38.858037 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:56:38.858018    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 11:56:38.867449 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:56:38.867426    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 11:57:13.148644 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.148612    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"]
Sep 10 11:57:13.151078 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.148985    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" containerName="console"
Sep 10 11:57:13.151078 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.148997    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" containerName="console"
Sep 10 11:57:13.151078 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.149051    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2466a590-c492-418e-b5c1-2bdc2dc49b1a" containerName="console"
Sep 10 11:57:13.152128 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.152113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.155011 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.154985    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 11:57:13.155334 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.155292    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 11:57:13.156083 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.156062    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-64jsz\""
Sep 10 11:57:13.156507 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.156491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 11:57:13.156614 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.156496    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 11:57:13.184370 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.184345    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"]
Sep 10 11:57:13.187923 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.187905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.188017 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.187950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.188017 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.187998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.188086 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.188017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkxt\" (UniqueName: \"kubernetes.io/projected/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-kube-api-access-dfkxt\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.188086 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.188038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.288861 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.288822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.289046 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.288903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.289046 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.288937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.289046 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.288953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkxt\" (UniqueName: \"kubernetes.io/projected/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-kube-api-access-dfkxt\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.289046 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.288974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.289381 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.289358    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-manifests\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.291605 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.291580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-tmp\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.291724 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.291707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.291797 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.291781    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-webhook-cert\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.304513 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.304485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkxt\" (UniqueName: \"kubernetes.io/projected/d2a3e6a9-9ae1-4180-aff2-a12e18b7411c-kube-api-access-dfkxt\") pod \"opendatahub-operator-controller-manager-d86dfbd7c-529mx\" (UID: \"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c\") " pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.463092 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.462997    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:13.746988 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.746899    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"]
Sep 10 11:57:13.750765 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:57:13.750737    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a3e6a9_9ae1_4180_aff2_a12e18b7411c.slice/crio-ebbbac13efcc2b8a75448475ed2db4ad0c605cbe23bb60fc31989f47c310c653 WatchSource:0}: Error finding container ebbbac13efcc2b8a75448475ed2db4ad0c605cbe23bb60fc31989f47c310c653: Status 404 returned error can't find the container with id ebbbac13efcc2b8a75448475ed2db4ad0c605cbe23bb60fc31989f47c310c653
Sep 10 11:57:13.752456 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:13.752440    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 11:57:14.426549 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:14.426503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx" event={"ID":"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c","Type":"ContainerStarted","Data":"ebbbac13efcc2b8a75448475ed2db4ad0c605cbe23bb60fc31989f47c310c653"}
Sep 10 11:57:17.442649 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:17.442613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx" event={"ID":"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c","Type":"ContainerStarted","Data":"2038b05faea2fe7bbd83809c2c1a8bc9b79e147c4c43b32d0dc23eafbd6e3184"}
Sep 10 11:57:18.448599 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:18.448565    2573 generic.go:356] "Generic (PLEG): container finished" podID="d2a3e6a9-9ae1-4180-aff2-a12e18b7411c" containerID="2038b05faea2fe7bbd83809c2c1a8bc9b79e147c4c43b32d0dc23eafbd6e3184" exitCode=0
Sep 10 11:57:18.448951 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:18.448650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx" event={"ID":"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c","Type":"ContainerDied","Data":"2038b05faea2fe7bbd83809c2c1a8bc9b79e147c4c43b32d0dc23eafbd6e3184"}
Sep 10 11:57:19.454441 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:19.454410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx" event={"ID":"d2a3e6a9-9ae1-4180-aff2-a12e18b7411c","Type":"ContainerStarted","Data":"b466f5210144ef0a83f575aa1908339f93b4d88afcbc083147b81908722c0862"}
Sep 10 11:57:19.454838 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:19.454489    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:19.495270 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:19.495215    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx" podStartSLOduration=1.96135378 podStartE2EDuration="6.49519823s" totalImagesPullingTime="3.468864205s" totalInitContainerRuntime="1.064980245s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:19.481440078 +0000 UTC m=+341.061367465" watchObservedRunningTime="2026-09-10 11:57:19.49519823 +0000 UTC m=+341.075125616"
Sep 10 11:57:30.462325 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:30.462230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d86dfbd7c-529mx"
Sep 10 11:57:41.505122 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.505088    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"]
Sep 10 11:57:41.508414 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.508393    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.512740 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.512707    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 11:57:41.512849 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.512760    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 11:57:41.512849 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.512776    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wcbp2\""
Sep 10 11:57:41.513776 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.513757    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 11:57:41.513875 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.513785    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 11:57:41.513875 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.513829    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 11:57:41.520961 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.520942    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"]
Sep 10 11:57:41.625665 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.625638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxdz\" (UniqueName: \"kubernetes.io/projected/83808e2b-aafd-4c4e-936f-94358b2fd12a-kube-api-access-gjxdz\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.625832 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.625676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.625832 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.625772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83808e2b-aafd-4c4e-936f-94358b2fd12a-manager-config\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.625832 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.625797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-metrics-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.726881 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.726845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.727054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.726915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83808e2b-aafd-4c4e-936f-94358b2fd12a-manager-config\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.727054 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.727024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-metrics-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.727173 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.727092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxdz\" (UniqueName: \"kubernetes.io/projected/83808e2b-aafd-4c4e-936f-94358b2fd12a-kube-api-access-gjxdz\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.727647 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.727616    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/83808e2b-aafd-4c4e-936f-94358b2fd12a-manager-config\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.729790 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.729769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.729870 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.729769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/83808e2b-aafd-4c4e-936f-94358b2fd12a-metrics-cert\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.737864 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.737841    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxdz\" (UniqueName: \"kubernetes.io/projected/83808e2b-aafd-4c4e-936f-94358b2fd12a-kube-api-access-gjxdz\") pod \"lws-controller-manager-5f7f7d474c-7vr5h\" (UID: \"83808e2b-aafd-4c4e-936f-94358b2fd12a\") " pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:41.817790 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:41.817722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:42.087089 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:42.087066    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"]
Sep 10 11:57:42.089256 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:57:42.089227    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83808e2b_aafd_4c4e_936f_94358b2fd12a.slice/crio-1c66fc6cffa2f185ba296dfc70556d1b83cd938b593732648ae43d9db76b9ff1 WatchSource:0}: Error finding container 1c66fc6cffa2f185ba296dfc70556d1b83cd938b593732648ae43d9db76b9ff1: Status 404 returned error can't find the container with id 1c66fc6cffa2f185ba296dfc70556d1b83cd938b593732648ae43d9db76b9ff1
Sep 10 11:57:42.563320 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:42.563268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h" event={"ID":"83808e2b-aafd-4c4e-936f-94358b2fd12a","Type":"ContainerStarted","Data":"1c66fc6cffa2f185ba296dfc70556d1b83cd938b593732648ae43d9db76b9ff1"}
Sep 10 11:57:43.111314 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.111266    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"]
Sep 10 11:57:43.114452 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.114433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.117737 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.117708    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 11:57:43.118236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.118216    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 11:57:43.118922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.118904    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gl5hz\""
Sep 10 11:57:43.132898 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.132867    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"]
Sep 10 11:57:43.241922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.241890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1a4ffee-5e55-4680-a6c3-76654dabf583-tmp\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.241922 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.241925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc5r\" (UniqueName: \"kubernetes.io/projected/b1a4ffee-5e55-4680-a6c3-76654dabf583-kube-api-access-mpc5r\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.242228 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.242030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a4ffee-5e55-4680-a6c3-76654dabf583-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.343125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.343091    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a4ffee-5e55-4680-a6c3-76654dabf583-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.343295 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.343160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1a4ffee-5e55-4680-a6c3-76654dabf583-tmp\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.343295 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.343181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc5r\" (UniqueName: \"kubernetes.io/projected/b1a4ffee-5e55-4680-a6c3-76654dabf583-kube-api-access-mpc5r\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.345646 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.345620    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1a4ffee-5e55-4680-a6c3-76654dabf583-tmp\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.345646 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.345637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a4ffee-5e55-4680-a6c3-76654dabf583-tls-certs\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.356955 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.356933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc5r\" (UniqueName: \"kubernetes.io/projected/b1a4ffee-5e55-4680-a6c3-76654dabf583-kube-api-access-mpc5r\") pod \"kube-auth-proxy-5986dd8d8b-znq8x\" (UID: \"b1a4ffee-5e55-4680-a6c3-76654dabf583\") " pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.425930 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.425902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"
Sep 10 11:57:43.723417 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:43.723372    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x"]
Sep 10 11:57:43.726423 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:57:43.726349    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a4ffee_5e55_4680_a6c3_76654dabf583.slice/crio-bf081b92ecef884ea866f7cb43332fc508c5636e0ea0888dbd4922a103e08520 WatchSource:0}: Error finding container bf081b92ecef884ea866f7cb43332fc508c5636e0ea0888dbd4922a103e08520: Status 404 returned error can't find the container with id bf081b92ecef884ea866f7cb43332fc508c5636e0ea0888dbd4922a103e08520
Sep 10 11:57:44.577249 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:44.577214    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x" event={"ID":"b1a4ffee-5e55-4680-a6c3-76654dabf583","Type":"ContainerStarted","Data":"bf081b92ecef884ea866f7cb43332fc508c5636e0ea0888dbd4922a103e08520"}
Sep 10 11:57:45.585572 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:45.585396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h" event={"ID":"83808e2b-aafd-4c4e-936f-94358b2fd12a","Type":"ContainerStarted","Data":"6ffdb7627a8243b58c35b8edc8bfeadfcb96f41c30bf3cfde14b45479c947f3a"}
Sep 10 11:57:45.586037 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:45.585801    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:57:45.607894 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:45.607801    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h" podStartSLOduration=1.522675253 podStartE2EDuration="4.607779232s" totalImagesPullingTime="3.085103979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:45.606581283 +0000 UTC m=+367.186508707" watchObservedRunningTime="2026-09-10 11:57:45.607779232 +0000 UTC m=+367.187706619"
Sep 10 11:57:47.598191 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:47.598105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x" event={"ID":"b1a4ffee-5e55-4680-a6c3-76654dabf583","Type":"ContainerStarted","Data":"72a3c0eb5c68fb513741221577b34849fffa292e653e3cdfb357271a587b2116"}
Sep 10 11:57:47.620331 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:47.620259    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5986dd8d8b-znq8x" podStartSLOduration=1.011731932 podStartE2EDuration="4.620242154s" totalImagesPullingTime="3.608510222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:57:47.61868 +0000 UTC m=+369.198607387" watchObservedRunningTime="2026-09-10 11:57:47.620242154 +0000 UTC m=+369.200169540"
Sep 10 11:57:56.593833 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:57:56.593800    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f7f7d474c-7vr5h"
Sep 10 11:59:14.278042 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.277959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"]
Sep 10 11:59:14.281788 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.281769    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.285886 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.285867    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 11:59:14.286021 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.285911    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cvsxm\""
Sep 10 11:59:14.290149 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.290130    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 11:59:14.290242 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.290232    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 11:59:14.290326 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.290231    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 11:59:14.294332 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.294297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"]
Sep 10 11:59:14.433429 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.433390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nhv\" (UniqueName: \"kubernetes.io/projected/8230adc6-bfca-47f4-a839-511a3173f664-kube-api-access-44nhv\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.433627 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.433514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8230adc6-bfca-47f4-a839-511a3173f664-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.433627 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.433549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8230adc6-bfca-47f4-a839-511a3173f664-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.534974 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.534894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8230adc6-bfca-47f4-a839-511a3173f664-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.534974 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.534927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8230adc6-bfca-47f4-a839-511a3173f664-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.535151 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.534978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nhv\" (UniqueName: \"kubernetes.io/projected/8230adc6-bfca-47f4-a839-511a3173f664-kube-api-access-44nhv\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.535715 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.535619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8230adc6-bfca-47f4-a839-511a3173f664-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.537523 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.537502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8230adc6-bfca-47f4-a839-511a3173f664-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.543917 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.543896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nhv\" (UniqueName: \"kubernetes.io/projected/8230adc6-bfca-47f4-a839-511a3173f664-kube-api-access-44nhv\") pod \"kuadrant-console-plugin-fbdd4f445-ddstq\" (UID: \"8230adc6-bfca-47f4-a839-511a3173f664\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.605771 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.605737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"
Sep 10 11:59:14.876991 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.876963    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq"]
Sep 10 11:59:14.879618 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:59:14.879596    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8230adc6_bfca_47f4_a839_511a3173f664.slice/crio-031556f1048e590ff5ac4b409b4e5f2b802cfb323cd0e2bc738678c8b0c59e47 WatchSource:0}: Error finding container 031556f1048e590ff5ac4b409b4e5f2b802cfb323cd0e2bc738678c8b0c59e47: Status 404 returned error can't find the container with id 031556f1048e590ff5ac4b409b4e5f2b802cfb323cd0e2bc738678c8b0c59e47
Sep 10 11:59:14.991979 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:14.991947    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq" event={"ID":"8230adc6-bfca-47f4-a839-511a3173f664","Type":"ContainerStarted","Data":"031556f1048e590ff5ac4b409b4e5f2b802cfb323cd0e2bc738678c8b0c59e47"}
Sep 10 11:59:21.023125 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:21.023092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq" event={"ID":"8230adc6-bfca-47f4-a839-511a3173f664","Type":"ContainerStarted","Data":"86b5c5d1b980218fb4c0ca633ee84a2fab66dfbb5090b1a3cb50247fa42bf055"}
Sep 10 11:59:21.047187 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:21.047113    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddstq" podStartSLOduration=1.9554501819999999 podStartE2EDuration="7.047060246s" totalImagesPullingTime="5.091610064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 11:59:21.046644267 +0000 UTC m=+462.626571653" watchObservedRunningTime="2026-09-10 11:59:21.047060246 +0000 UTC m=+462.626987632"
Sep 10 11:59:58.845388 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.845356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 11:59:58.869341 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.869284    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 11:59:58.869491 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.869399    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:58.872051 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.872034    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 11:59:58.928008 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.927979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb98r\" (UniqueName: \"kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:58.928171 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.928043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:58.939849 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:58.939812    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 11:59:59.028578 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.028542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:59.028792 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.028619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb98r\" (UniqueName: \"kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:59.029216 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.029192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:59.042420 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.042391    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb98r\" (UniqueName: \"kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r\") pod \"limitador-limitador-8588749977-wkcws\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") " pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:59.179131 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.179094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 11:59:59.474236 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.474156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 11:59:59.479395 ip-10-0-134-58 kubenswrapper[2573]: W0910 11:59:59.479365    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b07c2f4_669a_4d9d_b115_05be40dc41e2.slice/crio-b0be58d301281045e593fb1b6e4d112161794b96fef019c806e85ea70dfb3261 WatchSource:0}: Error finding container b0be58d301281045e593fb1b6e4d112161794b96fef019c806e85ea70dfb3261: Status 404 returned error can't find the container with id b0be58d301281045e593fb1b6e4d112161794b96fef019c806e85ea70dfb3261
Sep 10 11:59:59.940954 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.940922    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 11:59:59.945607 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.945586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 11:59:59.948102 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.948085    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m6mmm\""
Sep 10 11:59:59.951352 ip-10-0-134-58 kubenswrapper[2573]: I0910 11:59:59.951330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 12:00:00.039598 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.039550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gwn\" (UniqueName: \"kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn\") pod \"authorino-59c974bd48-dn4q2\" (UID: \"0b578a42-6fcc-4ff7-ad78-0a56b3198562\") " pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:00.140738 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.140699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gwn\" (UniqueName: \"kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn\") pod \"authorino-59c974bd48-dn4q2\" (UID: \"0b578a42-6fcc-4ff7-ad78-0a56b3198562\") " pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:00.171359 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.171321    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gwn\" (UniqueName: \"kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn\") pod \"authorino-59c974bd48-dn4q2\" (UID: \"0b578a42-6fcc-4ff7-ad78-0a56b3198562\") " pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:00.195212 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.195128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" event={"ID":"0b07c2f4-669a-4d9d-b115-05be40dc41e2","Type":"ContainerStarted","Data":"b0be58d301281045e593fb1b6e4d112161794b96fef019c806e85ea70dfb3261"}
Sep 10 12:00:00.255974 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.255938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:00.545746 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:00.545605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 12:00:00.548269 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:00:00.548235    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b578a42_6fcc_4ff7_ad78_0a56b3198562.slice/crio-d6ee201746ebdc5b85653ef0c6aed69878eb473ab7c9aa7aa84d62a3e07ebcc2 WatchSource:0}: Error finding container d6ee201746ebdc5b85653ef0c6aed69878eb473ab7c9aa7aa84d62a3e07ebcc2: Status 404 returned error can't find the container with id d6ee201746ebdc5b85653ef0c6aed69878eb473ab7c9aa7aa84d62a3e07ebcc2
Sep 10 12:00:01.201399 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:01.201365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dn4q2" event={"ID":"0b578a42-6fcc-4ff7-ad78-0a56b3198562","Type":"ContainerStarted","Data":"d6ee201746ebdc5b85653ef0c6aed69878eb473ab7c9aa7aa84d62a3e07ebcc2"}
Sep 10 12:00:02.207265 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:02.207232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" event={"ID":"0b07c2f4-669a-4d9d-b115-05be40dc41e2","Type":"ContainerStarted","Data":"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"}
Sep 10 12:00:02.207658 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:02.207336    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 12:00:02.226014 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:02.225971    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" podStartSLOduration=1.871933711 podStartE2EDuration="4.225955289s" totalImagesPullingTime="2.354021578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:02.224414823 +0000 UTC m=+503.804342210" watchObservedRunningTime="2026-09-10 12:00:02.225955289 +0000 UTC m=+503.805882674"
Sep 10 12:00:04.219674 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:04.219632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dn4q2" event={"ID":"0b578a42-6fcc-4ff7-ad78-0a56b3198562","Type":"ContainerStarted","Data":"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"}
Sep 10 12:00:04.235275 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:04.235194    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dn4q2" podStartSLOduration=1.793401619 podStartE2EDuration="5.235175768s" totalImagesPullingTime="3.441774149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 11:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:04.234972117 +0000 UTC m=+505.814899503" watchObservedRunningTime="2026-09-10 12:00:04.235175768 +0000 UTC m=+505.815103156"
Sep 10 12:00:12.890791 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:12.890754    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 12:00:12.891199 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:12.890950    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" podUID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" containerName="limitador" containerID="cri-o://0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c" gracePeriod=30
Sep 10 12:00:12.891774 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:12.891694    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 12:00:13.441212 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.441188    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 12:00:13.563943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.563861    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file\") pod \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") "
Sep 10 12:00:13.563943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.563921    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb98r\" (UniqueName: \"kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r\") pod \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\" (UID: \"0b07c2f4-669a-4d9d-b115-05be40dc41e2\") "
Sep 10 12:00:13.564186 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.564162    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file" (OuterVolumeSpecName: "config-file") pod "0b07c2f4-669a-4d9d-b115-05be40dc41e2" (UID: "0b07c2f4-669a-4d9d-b115-05be40dc41e2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:00:13.566154 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.566137    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r" (OuterVolumeSpecName: "kube-api-access-vb98r") pod "0b07c2f4-669a-4d9d-b115-05be40dc41e2" (UID: "0b07c2f4-669a-4d9d-b115-05be40dc41e2"). InnerVolumeSpecName "kube-api-access-vb98r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:00:13.665528 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.665502    2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0b07c2f4-669a-4d9d-b115-05be40dc41e2-config-file\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:13.665528 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:13.665528    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb98r\" (UniqueName: \"kubernetes.io/projected/0b07c2f4-669a-4d9d-b115-05be40dc41e2-kube-api-access-vb98r\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:14.263492 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.263456    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" containerID="0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c" exitCode=0
Sep 10 12:00:14.263913 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.263500    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-wkcws"
Sep 10 12:00:14.263913 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.263496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" event={"ID":"0b07c2f4-669a-4d9d-b115-05be40dc41e2","Type":"ContainerDied","Data":"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"}
Sep 10 12:00:14.263913 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.263613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-wkcws" event={"ID":"0b07c2f4-669a-4d9d-b115-05be40dc41e2","Type":"ContainerDied","Data":"b0be58d301281045e593fb1b6e4d112161794b96fef019c806e85ea70dfb3261"}
Sep 10 12:00:14.263913 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.263635    2573 scope.go:117] "RemoveContainer" containerID="0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"
Sep 10 12:00:14.272255 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.272238    2573 scope.go:117] "RemoveContainer" containerID="0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"
Sep 10 12:00:14.272542 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:00:14.272525    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c\": container with ID starting with 0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c not found: ID does not exist" containerID="0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"
Sep 10 12:00:14.272583 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.272551    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c"} err="failed to get container status \"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c\": rpc error: code = NotFound desc = could not find container \"0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c\": container with ID starting with 0c6c91f68bfb6eda69c4bf7cb8af96f10c24aa7718677232a2f5d0c789c7e14c not found: ID does not exist"
Sep 10 12:00:14.286871 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.286849    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 12:00:14.291226 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.291207    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-wkcws"]
Sep 10 12:00:14.946448 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:14.946418    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" path="/var/lib/kubelet/pods/0b07c2f4-669a-4d9d-b115-05be40dc41e2/volumes"
Sep 10 12:00:15.583852 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.583820    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"]
Sep 10 12:00:15.584437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.584415    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" containerName="limitador"
Sep 10 12:00:15.584495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.584441    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" containerName="limitador"
Sep 10 12:00:15.584561 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.584550    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b07c2f4-669a-4d9d-b115-05be40dc41e2" containerName="limitador"
Sep 10 12:00:15.589312 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.589276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.592053 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.592032    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 12:00:15.592157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.592034    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vhvf7\""
Sep 10 12:00:15.592157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.592075    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 12:00:15.592900 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.592885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:00:15.595735 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.595713    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"]
Sep 10 12:00:15.684836 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.684799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5f04697-fb53-463e-9a01-d4876999dd2c-data\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.685016 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.684868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzbz\" (UniqueName: \"kubernetes.io/projected/f5f04697-fb53-463e-9a01-d4876999dd2c-kube-api-access-tmzbz\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.785330 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.785271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5f04697-fb53-463e-9a01-d4876999dd2c-data\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.785513 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.785352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzbz\" (UniqueName: \"kubernetes.io/projected/f5f04697-fb53-463e-9a01-d4876999dd2c-kube-api-access-tmzbz\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.785717 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.785694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5f04697-fb53-463e-9a01-d4876999dd2c-data\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.794930 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.794908    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzbz\" (UniqueName: \"kubernetes.io/projected/f5f04697-fb53-463e-9a01-d4876999dd2c-kube-api-access-tmzbz\") pod \"postgres-69fb4594cf-ljxxw\" (UID: \"f5f04697-fb53-463e-9a01-d4876999dd2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:15.900831 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:15.900806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:16.175046 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:16.175019    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"]
Sep 10 12:00:16.177428 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:00:16.177400    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f04697_fb53_463e_9a01_d4876999dd2c.slice/crio-efe246cb2137d885bed2dd0f8b19339e8cd418be1bcc6f45b83d98eb9b53cdcb WatchSource:0}: Error finding container efe246cb2137d885bed2dd0f8b19339e8cd418be1bcc6f45b83d98eb9b53cdcb: Status 404 returned error can't find the container with id efe246cb2137d885bed2dd0f8b19339e8cd418be1bcc6f45b83d98eb9b53cdcb
Sep 10 12:00:16.275572 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:16.275545    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw" event={"ID":"f5f04697-fb53-463e-9a01-d4876999dd2c","Type":"ContainerStarted","Data":"efe246cb2137d885bed2dd0f8b19339e8cd418be1bcc6f45b83d98eb9b53cdcb"}
Sep 10 12:00:20.298029 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:20.297994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw" event={"ID":"f5f04697-fb53-463e-9a01-d4876999dd2c","Type":"ContainerStarted","Data":"12409d1344a04152c6adee8fc54ad736ff08f9fc00512a6f2e9486e042bf5ddc"}
Sep 10 12:00:20.298414 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:20.298219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:20.316797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:20.316728    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw" podStartSLOduration=2.116802992 podStartE2EDuration="5.316708498s" totalImagesPullingTime="3.199905506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:20.314989035 +0000 UTC m=+521.894916420" watchObservedRunningTime="2026-09-10 12:00:20.316708498 +0000 UTC m=+521.896635886"
Sep 10 12:00:26.332144 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.332114    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ljxxw"
Sep 10 12:00:26.903933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.903896    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-tp8sh"]
Sep 10 12:00:26.911077 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.911056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:26.915384 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.915359    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-tp8sh"]
Sep 10 12:00:26.971053 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.971021    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tp8sh"]
Sep 10 12:00:26.971287 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:00:26.971271    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-75ccl], unattached volumes=[], failed to process volumes=[kube-api-access-75ccl]: context canceled" pod="kuadrant-system/authorino-557f876f69-tp8sh" podUID="014e374c-e0bf-4aa5-947c-be4febb1f5ee"
Sep 10 12:00:26.980915 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.980892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ccl\" (UniqueName: \"kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl\") pod \"authorino-557f876f69-tp8sh\" (UID: \"014e374c-e0bf-4aa5-947c-be4febb1f5ee\") " pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:26.997850 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:26.997823    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pt674"]
Sep 10 12:00:27.000682 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.000666    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.003479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.003458    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 12:00:27.008619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.008599    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pt674"]
Sep 10 12:00:27.082076 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.082044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nxj\" (UniqueName: \"kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.082230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.082155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ccl\" (UniqueName: \"kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl\") pod \"authorino-557f876f69-tp8sh\" (UID: \"014e374c-e0bf-4aa5-947c-be4febb1f5ee\") " pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:27.082230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.082186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.091045 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.091013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ccl\" (UniqueName: \"kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl\") pod \"authorino-557f876f69-tp8sh\" (UID: \"014e374c-e0bf-4aa5-947c-be4febb1f5ee\") " pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:27.152489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.152454    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pt674"]
Sep 10 12:00:27.152731 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:00:27.152714    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-44nxj tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pt674" podUID="b316446a-f69d-4ad3-a3ba-e646b5931894"
Sep 10 12:00:27.180433 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.180379    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:00:27.183067 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.183046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.183136 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.183085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nxj\" (UniqueName: \"kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.183398 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.183384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.185498 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.185474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.196329 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.195190    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:00:27.196329 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.196108    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nxj\" (UniqueName: \"kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj\") pod \"authorino-6d4dfc649f-pt674\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") " pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.283794 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.283769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.283916 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.283801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkt6\" (UniqueName: \"kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.331115 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.331089    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.331115 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.331095    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:27.337254 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.337234    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:27.340444 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.340428    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:27.384398 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.384374    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44nxj\" (UniqueName: \"kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj\") pod \"b316446a-f69d-4ad3-a3ba-e646b5931894\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") "
Sep 10 12:00:27.384491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.384444    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert\") pod \"b316446a-f69d-4ad3-a3ba-e646b5931894\" (UID: \"b316446a-f69d-4ad3-a3ba-e646b5931894\") "
Sep 10 12:00:27.384491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.384472    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75ccl\" (UniqueName: \"kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl\") pod \"014e374c-e0bf-4aa5-947c-be4febb1f5ee\" (UID: \"014e374c-e0bf-4aa5-947c-be4febb1f5ee\") "
Sep 10 12:00:27.384691 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.384669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.384817 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.384713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkt6\" (UniqueName: \"kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.386733 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.386705    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b316446a-f69d-4ad3-a3ba-e646b5931894" (UID: "b316446a-f69d-4ad3-a3ba-e646b5931894"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:00:27.386733 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.386722    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj" (OuterVolumeSpecName: "kube-api-access-44nxj") pod "b316446a-f69d-4ad3-a3ba-e646b5931894" (UID: "b316446a-f69d-4ad3-a3ba-e646b5931894"). InnerVolumeSpecName "kube-api-access-44nxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:00:27.386887 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.386817    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl" (OuterVolumeSpecName: "kube-api-access-75ccl") pod "014e374c-e0bf-4aa5-947c-be4febb1f5ee" (UID: "014e374c-e0bf-4aa5-947c-be4febb1f5ee"). InnerVolumeSpecName "kube-api-access-75ccl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:00:27.387211 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.387189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.393269 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.393119    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkt6\" (UniqueName: \"kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6\") pod \"authorino-76dc7cc46-7qdgc\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") " pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.485536 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.485459    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44nxj\" (UniqueName: \"kubernetes.io/projected/b316446a-f69d-4ad3-a3ba-e646b5931894-kube-api-access-44nxj\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:27.485536 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.485486    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b316446a-f69d-4ad3-a3ba-e646b5931894-tls-cert\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:27.485536 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.485496    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ccl\" (UniqueName: \"kubernetes.io/projected/014e374c-e0bf-4aa5-947c-be4febb1f5ee-kube-api-access-75ccl\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:27.512429 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.512402    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:00:27.779339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:27.779170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:00:27.781857 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:00:27.781832    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57b26d70_cbb6_4035_ba1a_5766768666ab.slice/crio-760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e WatchSource:0}: Error finding container 760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e: Status 404 returned error can't find the container with id 760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e
Sep 10 12:00:28.336623 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.336593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" event={"ID":"57b26d70-cbb6-4035-ba1a-5766768666ab","Type":"ContainerStarted","Data":"760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e"}
Sep 10 12:00:28.336798 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.336629    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-tp8sh"
Sep 10 12:00:28.336798 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.336633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pt674"
Sep 10 12:00:28.369340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.369297    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pt674"]
Sep 10 12:00:28.372732 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.372708    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pt674"]
Sep 10 12:00:28.393891 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.393867    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-tp8sh"]
Sep 10 12:00:28.397416 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.397394    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-tp8sh"]
Sep 10 12:00:28.948280 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.948189    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014e374c-e0bf-4aa5-947c-be4febb1f5ee" path="/var/lib/kubelet/pods/014e374c-e0bf-4aa5-947c-be4febb1f5ee/volumes"
Sep 10 12:00:28.948642 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:28.948622    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b316446a-f69d-4ad3-a3ba-e646b5931894" path="/var/lib/kubelet/pods/b316446a-f69d-4ad3-a3ba-e646b5931894/volumes"
Sep 10 12:00:29.342832 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.342797    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" event={"ID":"57b26d70-cbb6-4035-ba1a-5766768666ab","Type":"ContainerStarted","Data":"35551bac1f8d8bf481843042a5c096babb59118202a263af87c715bf01399abc"}
Sep 10 12:00:29.360816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.360758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" podStartSLOduration=1.338889399 podStartE2EDuration="2.360741965s" totalImagesPullingTime="1.021852566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:00:29.35967748 +0000 UTC m=+530.939604867" watchObservedRunningTime="2026-09-10 12:00:29.360741965 +0000 UTC m=+530.940669387"
Sep 10 12:00:29.389272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.389233    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 12:00:29.389728 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.389436    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dn4q2" podUID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" containerName="authorino" containerID="cri-o://87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487" gracePeriod=30
Sep 10 12:00:29.641340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.641316    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:29.702684 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.702652    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89gwn\" (UniqueName: \"kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn\") pod \"0b578a42-6fcc-4ff7-ad78-0a56b3198562\" (UID: \"0b578a42-6fcc-4ff7-ad78-0a56b3198562\") "
Sep 10 12:00:29.704755 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.704734    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn" (OuterVolumeSpecName: "kube-api-access-89gwn") pod "0b578a42-6fcc-4ff7-ad78-0a56b3198562" (UID: "0b578a42-6fcc-4ff7-ad78-0a56b3198562"). InnerVolumeSpecName "kube-api-access-89gwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:00:29.804030 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:29.803961    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gwn\" (UniqueName: \"kubernetes.io/projected/0b578a42-6fcc-4ff7-ad78-0a56b3198562-kube-api-access-89gwn\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:00:30.348327 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.348270    2573 generic.go:356] "Generic (PLEG): container finished" podID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" containerID="87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487" exitCode=0
Sep 10 12:00:30.348327 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.348331    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dn4q2"
Sep 10 12:00:30.348523 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.348328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dn4q2" event={"ID":"0b578a42-6fcc-4ff7-ad78-0a56b3198562","Type":"ContainerDied","Data":"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"}
Sep 10 12:00:30.348523 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.348443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dn4q2" event={"ID":"0b578a42-6fcc-4ff7-ad78-0a56b3198562","Type":"ContainerDied","Data":"d6ee201746ebdc5b85653ef0c6aed69878eb473ab7c9aa7aa84d62a3e07ebcc2"}
Sep 10 12:00:30.348523 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.348466    2573 scope.go:117] "RemoveContainer" containerID="87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"
Sep 10 12:00:30.359775 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.359751    2573 scope.go:117] "RemoveContainer" containerID="87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"
Sep 10 12:00:30.360093 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:00:30.360076    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487\": container with ID starting with 87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487 not found: ID does not exist" containerID="87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"
Sep 10 12:00:30.360141 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.360102    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487"} err="failed to get container status \"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487\": rpc error: code = NotFound desc = could not find container \"87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487\": container with ID starting with 87ca0148e1bc988ec4464ab30053e86616807933aa3c07850ab62cbeb5a5f487 not found: ID does not exist"
Sep 10 12:00:30.372119 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.372093    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 12:00:30.376128 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.376109    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dn4q2"]
Sep 10 12:00:30.952045 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:00:30.952009    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" path="/var/lib/kubelet/pods/0b578a42-6fcc-4ff7-ad78-0a56b3198562/volumes"
Sep 10 12:01:16.371070 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.371028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"]
Sep 10 12:01:16.371659 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.371637    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" containerName="authorino"
Sep 10 12:01:16.371705 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.371669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" containerName="authorino"
Sep 10 12:01:16.371812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.371799    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b578a42-6fcc-4ff7-ad78-0a56b3198562" containerName="authorino"
Sep 10 12:01:16.374330 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.374290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.376909 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.376881    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 12:01:16.377365 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.377342    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r8gmk\""
Sep 10 12:01:16.389624 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.389591    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"]
Sep 10 12:01:16.505656 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.505624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f851025f-f34e-4ef8-8ea6-93a31b66a242-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.505821 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.505666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts8j\" (UniqueName: \"kubernetes.io/projected/f851025f-f34e-4ef8-8ea6-93a31b66a242-kube-api-access-tts8j\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.606574 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.606518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f851025f-f34e-4ef8-8ea6-93a31b66a242-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.606574 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.606563    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts8j\" (UniqueName: \"kubernetes.io/projected/f851025f-f34e-4ef8-8ea6-93a31b66a242-kube-api-access-tts8j\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.607232 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.607208    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f851025f-f34e-4ef8-8ea6-93a31b66a242-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.616435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.616402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts8j\" (UniqueName: \"kubernetes.io/projected/f851025f-f34e-4ef8-8ea6-93a31b66a242-kube-api-access-tts8j\") pod \"payload-pre-processing-59fbc47bf8-47wtr\" (UID: \"f851025f-f34e-4ef8-8ea6-93a31b66a242\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.685672 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.685634    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:16.960517 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:16.960496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"]
Sep 10 12:01:16.962484 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:01:16.962454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf851025f_f34e_4ef8_8ea6_93a31b66a242.slice/crio-150541204a66cc8f432df186c2d3015fae23b6fc8e9d40213e9d6ed87a239ffe WatchSource:0}: Error finding container 150541204a66cc8f432df186c2d3015fae23b6fc8e9d40213e9d6ed87a239ffe: Status 404 returned error can't find the container with id 150541204a66cc8f432df186c2d3015fae23b6fc8e9d40213e9d6ed87a239ffe
Sep 10 12:01:17.565081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:17.565044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr" event={"ID":"f851025f-f34e-4ef8-8ea6-93a31b66a242","Type":"ContainerStarted","Data":"150541204a66cc8f432df186c2d3015fae23b6fc8e9d40213e9d6ed87a239ffe"}
Sep 10 12:01:20.581776 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:20.581737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr" event={"ID":"f851025f-f34e-4ef8-8ea6-93a31b66a242","Type":"ContainerStarted","Data":"c5d28fe17f50fa33680c919457cc283f5eea144f1c890a585d7cd4032808cc05"}
Sep 10 12:01:20.582140 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:20.581865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:20.603486 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:20.603444    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr" podStartSLOduration=1.9786457689999999 podStartE2EDuration="4.603430209s" totalImagesPullingTime="2.62478444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:01:20.602947613 +0000 UTC m=+582.182875000" watchObservedRunningTime="2026-09-10 12:01:20.603430209 +0000 UTC m=+582.183357619"
Sep 10 12:01:29.882525 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.882489    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"]
Sep 10 12:01:29.886746 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.886720    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.889797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.889771    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 12:01:29.889909 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.889809    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 12:01:29.890750 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.890730    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 12:01:29.890884 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.890862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wfrs7\""
Sep 10 12:01:29.896458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.896435    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"]
Sep 10 12:01:29.927640 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.927640 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.927860 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.927860 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdlm\" (UniqueName: \"kubernetes.io/projected/7aca7ba3-7f84-4850-a4af-d869a9fd9161-kube-api-access-8tdlm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.927965 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:29.927965 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:29.927922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028389 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028389 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028644 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028644 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028644 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028467    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028644 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdlm\" (UniqueName: \"kubernetes.io/projected/7aca7ba3-7f84-4850-a4af-d869a9fd9161-kube-api-access-8tdlm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028873 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.028922 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.029012 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.028990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.031121 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.031097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aca7ba3-7f84-4850-a4af-d869a9fd9161-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.031346 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.031328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aca7ba3-7f84-4850-a4af-d869a9fd9161-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.041833 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.041811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdlm\" (UniqueName: \"kubernetes.io/projected/7aca7ba3-7f84-4850-a4af-d869a9fd9161-kube-api-access-8tdlm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82\" (UID: \"7aca7ba3-7f84-4850-a4af-d869a9fd9161\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.198535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.198504    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:30.485002 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.484820    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"]
Sep 10 12:01:30.487354 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:01:30.487316    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aca7ba3_7f84_4850_a4af_d869a9fd9161.slice/crio-92d0bc394c82450b5f0310d5b495d008ae42a8ffaa29f2915c6e79159535f752 WatchSource:0}: Error finding container 92d0bc394c82450b5f0310d5b495d008ae42a8ffaa29f2915c6e79159535f752: Status 404 returned error can't find the container with id 92d0bc394c82450b5f0310d5b495d008ae42a8ffaa29f2915c6e79159535f752
Sep 10 12:01:30.630820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:30.630788    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82" event={"ID":"7aca7ba3-7f84-4850-a4af-d869a9fd9161","Type":"ContainerStarted","Data":"92d0bc394c82450b5f0310d5b495d008ae42a8ffaa29f2915c6e79159535f752"}
Sep 10 12:01:31.589884 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:31.589853    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-47wtr"
Sep 10 12:01:32.643368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:32.643331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82" event={"ID":"7aca7ba3-7f84-4850-a4af-d869a9fd9161","Type":"ContainerStarted","Data":"4a79affae412fc7e4c06f1859523e232ff0090011f287d55cc4846a43942b2fb"}
Sep 10 12:01:32.661837 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:32.661789    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82" podStartSLOduration=2.455967862 podStartE2EDuration="3.661773619s" totalImagesPullingTime="1.205805757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:01:32.661070498 +0000 UTC m=+594.240997884" watchObservedRunningTime="2026-09-10 12:01:32.661773619 +0000 UTC m=+594.241701005"
Sep 10 12:01:38.891940 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:38.891913    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:01:38.892399 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:38.891985    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:01:40.199371 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:40.199335    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:40.199773 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:40.199402    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:40.212068 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:40.212040    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:40.691654 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:40.691615    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82"
Sep 10 12:01:45.260535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.260502    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"]
Sep 10 12:01:45.265019 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.265000    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.268478 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.268449    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 12:01:45.274777 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.274758    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"]
Sep 10 12:01:45.366825 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.366775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.366825 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.366834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.367039 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.366950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.367039 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.366999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.367106 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.367046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.367106 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.367065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6787s\" (UniqueName: \"kubernetes.io/projected/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-kube-api-access-6787s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468170 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468170 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468173    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468442 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468442 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468442 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468442 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6787s\" (UniqueName: \"kubernetes.io/projected/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-kube-api-access-6787s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468738 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468873 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.468873 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.468858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.470742 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.470714    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.470907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.470889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.477610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.477589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6787s\" (UniqueName: \"kubernetes.io/projected/c715f9c4-4d08-42f4-a186-9aa60c0bcdd3-kube-api-access-6787s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-np2t9\" (UID: \"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.575799 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.575719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:45.850627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:45.850549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"]
Sep 10 12:01:45.854363 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:01:45.854333    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc715f9c4_4d08_42f4_a186_9aa60c0bcdd3.slice/crio-eadeef98f87a3a5b3d94c27745ba2918ea1155e49d16d81c2aad296c4f024d47 WatchSource:0}: Error finding container eadeef98f87a3a5b3d94c27745ba2918ea1155e49d16d81c2aad296c4f024d47: Status 404 returned error can't find the container with id eadeef98f87a3a5b3d94c27745ba2918ea1155e49d16d81c2aad296c4f024d47
Sep 10 12:01:46.711161 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:46.711128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9" event={"ID":"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3","Type":"ContainerStarted","Data":"64341fcb1bfdb9c8713ed694adb1b1d70a451989c928f481adbed68efa80a7b4"}
Sep 10 12:01:46.711598 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:46.711180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9" event={"ID":"c715f9c4-4d08-42f4-a186-9aa60c0bcdd3","Type":"ContainerStarted","Data":"eadeef98f87a3a5b3d94c27745ba2918ea1155e49d16d81c2aad296c4f024d47"}
Sep 10 12:01:46.732153 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:46.732096    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9" podStartSLOduration=1.5696526309999999 podStartE2EDuration="1.732080198s" totalImagesPullingTime="162.427567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:01:46.730611548 +0000 UTC m=+608.310538933" watchObservedRunningTime="2026-09-10 12:01:46.732080198 +0000 UTC m=+608.312007583"
Sep 10 12:01:55.575898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:55.575861    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:55.576530 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:55.576165    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:55.591028 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:55.591003    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:01:55.767936 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:01:55.767903    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-np2t9"
Sep 10 12:02:42.157535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.157502    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"]
Sep 10 12:02:42.160850 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.160825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.163602 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.163579    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 12:02:42.170780 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.170757    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"]
Sep 10 12:02:42.283427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqghs\" (UniqueName: \"kubernetes.io/projected/13ea1a33-13c8-4859-9266-827d9d06445f-kube-api-access-gqghs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.283593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ea1a33-13c8-4859-9266-827d9d06445f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.283593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.283593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.283694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.283694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.283631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384494 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ea1a33-13c8-4859-9266-827d9d06445f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384494 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384726 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384726 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384726 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384726 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqghs\" (UniqueName: \"kubernetes.io/projected/13ea1a33-13c8-4859-9266-827d9d06445f-kube-api-access-gqghs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.384976 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.385041 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.384997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.385091 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.385039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.387104 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.387088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13ea1a33-13c8-4859-9266-827d9d06445f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.387273 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.387258    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13ea1a33-13c8-4859-9266-827d9d06445f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.394664 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.394632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqghs\" (UniqueName: \"kubernetes.io/projected/13ea1a33-13c8-4859-9266-827d9d06445f-kube-api-access-gqghs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44\" (UID: \"13ea1a33-13c8-4859-9266-827d9d06445f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.472683 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.472578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:42.757042 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.754462    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"]
Sep 10 12:02:42.762033 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.762007    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:02:42.977973 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:42.977946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44" event={"ID":"13ea1a33-13c8-4859-9266-827d9d06445f","Type":"ContainerStarted","Data":"684ad88c364509d126b4da9bae2c92492415e55e39d9efd43cc869be57441e88"}
Sep 10 12:02:43.984784 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:43.984736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44" event={"ID":"13ea1a33-13c8-4859-9266-827d9d06445f","Type":"ContainerStarted","Data":"d7b7566aaed20a87c561343f287f63c6e192d7711adb929d6ac0574c51b2b062"}
Sep 10 12:02:44.010497 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:44.010435    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44" podStartSLOduration=1.839348118 podStartE2EDuration="2.010414406s" totalImagesPullingTime="171.066288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:02:44.008837814 +0000 UTC m=+665.588765199" watchObservedRunningTime="2026-09-10 12:02:44.010414406 +0000 UTC m=+665.590341795"
Sep 10 12:02:52.472831 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:52.472774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:52.473251 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:52.472863    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:52.485243 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:52.485221    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:02:53.036539 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:02:53.036512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44"
Sep 10 12:04:18.200518 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.200480    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:04:18.201279 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.201247    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" podUID="57b26d70-cbb6-4035-ba1a-5766768666ab" containerName="authorino" containerID="cri-o://35551bac1f8d8bf481843042a5c096babb59118202a263af87c715bf01399abc" gracePeriod=30
Sep 10 12:04:18.435215 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.435185    2573 generic.go:356] "Generic (PLEG): container finished" podID="57b26d70-cbb6-4035-ba1a-5766768666ab" containerID="35551bac1f8d8bf481843042a5c096babb59118202a263af87c715bf01399abc" exitCode=0
Sep 10 12:04:18.435382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.435234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" event={"ID":"57b26d70-cbb6-4035-ba1a-5766768666ab","Type":"ContainerDied","Data":"35551bac1f8d8bf481843042a5c096babb59118202a263af87c715bf01399abc"}
Sep 10 12:04:18.435382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.435277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76dc7cc46-7qdgc" event={"ID":"57b26d70-cbb6-4035-ba1a-5766768666ab","Type":"ContainerDied","Data":"760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e"}
Sep 10 12:04:18.435382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.435289    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760456b09c53d42dc757d6e5954c6c6fb928d28177d0da96193c89ddd01d8f4e"
Sep 10 12:04:18.441702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.441685    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:04:18.468142 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.468089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert\") pod \"57b26d70-cbb6-4035-ba1a-5766768666ab\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") "
Sep 10 12:04:18.468222 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.468202    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmkt6\" (UniqueName: \"kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6\") pod \"57b26d70-cbb6-4035-ba1a-5766768666ab\" (UID: \"57b26d70-cbb6-4035-ba1a-5766768666ab\") "
Sep 10 12:04:18.470260 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.470229    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6" (OuterVolumeSpecName: "kube-api-access-nmkt6") pod "57b26d70-cbb6-4035-ba1a-5766768666ab" (UID: "57b26d70-cbb6-4035-ba1a-5766768666ab"). InnerVolumeSpecName "kube-api-access-nmkt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:04:18.478739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.478716    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "57b26d70-cbb6-4035-ba1a-5766768666ab" (UID: "57b26d70-cbb6-4035-ba1a-5766768666ab"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:04:18.569522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.569483    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57b26d70-cbb6-4035-ba1a-5766768666ab-tls-cert\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:04:18.569522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:18.569513    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmkt6\" (UniqueName: \"kubernetes.io/projected/57b26d70-cbb6-4035-ba1a-5766768666ab-kube-api-access-nmkt6\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:04:19.440806 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:19.440766    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76dc7cc46-7qdgc"
Sep 10 12:04:19.465246 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:19.465218    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:04:19.468440 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:19.468415    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76dc7cc46-7qdgc"]
Sep 10 12:04:20.946640 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:20.946608    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b26d70-cbb6-4035-ba1a-5766768666ab" path="/var/lib/kubelet/pods/57b26d70-cbb6-4035-ba1a-5766768666ab/volumes"
Sep 10 12:04:38.138676 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.138638    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7967-139929861815728"]
Sep 10 12:04:38.139069 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.139020    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b26d70-cbb6-4035-ba1a-5766768666ab" containerName="authorino"
Sep 10 12:04:38.139069 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.139032    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b26d70-cbb6-4035-ba1a-5766768666ab" containerName="authorino"
Sep 10 12:04:38.139152 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.139096    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b26d70-cbb6-4035-ba1a-5766768666ab" containerName="authorino"
Sep 10 12:04:38.142193 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.142177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:38.147785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.147765    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7cc6f\""
Sep 10 12:04:38.160197 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.160173    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"]
Sep 10 12:04:38.251077 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.251046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjml7\" (UniqueName: \"kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7\") pod \"test-curl-7967-139929861815728\" (UID: \"5b50d5f7-fbba-4ae4-be53-deed51a9f5be\") " pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:38.351873 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.351837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjml7\" (UniqueName: \"kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7\") pod \"test-curl-7967-139929861815728\" (UID: \"5b50d5f7-fbba-4ae4-be53-deed51a9f5be\") " pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:38.366273 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.366242    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjml7\" (UniqueName: \"kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7\") pod \"test-curl-7967-139929861815728\" (UID: \"5b50d5f7-fbba-4ae4-be53-deed51a9f5be\") " pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:38.450626 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.450597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:38.725220 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:38.725197    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"]
Sep 10 12:04:38.727805 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:38.727772    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b50d5f7_fbba_4ae4_be53_deed51a9f5be.slice/crio-e0f33139722a193d25986ffb2aa96d70e35de91d6b592d5dd3b3624aea6ff6d1 WatchSource:0}: Error finding container e0f33139722a193d25986ffb2aa96d70e35de91d6b592d5dd3b3624aea6ff6d1: Status 404 returned error can't find the container with id e0f33139722a193d25986ffb2aa96d70e35de91d6b592d5dd3b3624aea6ff6d1
Sep 10 12:04:39.542430 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:39.542388    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7967-139929861815728" event={"ID":"5b50d5f7-fbba-4ae4-be53-deed51a9f5be","Type":"ContainerStarted","Data":"e0f33139722a193d25986ffb2aa96d70e35de91d6b592d5dd3b3624aea6ff6d1"}
Sep 10 12:04:41.365413 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.365384    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:04:41.368594 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.368577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.371732 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.371708    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-89d91a-dockercfg-x52zv\""
Sep 10 12:04:41.372015 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.371998    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-89d91a\""
Sep 10 12:04:41.386443 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.386423    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:04:41.444969 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.444935    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:04:41.448620 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.448590    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.458747 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.458726    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:04:41.481288 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.481261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.481416 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.481293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lqzs\" (UniqueName: \"kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.556346 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.556314    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b50d5f7-fbba-4ae4-be53-deed51a9f5be" containerID="0bddfa81549d50dbd99c48d65e48a1e931ef867e7caaaccaf5775d1a2659eff3" exitCode=0
Sep 10 12:04:41.556458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.556334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7967-139929861815728" event={"ID":"5b50d5f7-fbba-4ae4-be53-deed51a9f5be","Type":"ContainerDied","Data":"0bddfa81549d50dbd99c48d65e48a1e931ef867e7caaaccaf5775d1a2659eff3"}
Sep 10 12:04:41.582080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.582056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.582394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.582089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lqzs\" (UniqueName: \"kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.582394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.582324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.582528 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.582433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndz2s\" (UniqueName: \"kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.582801 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.582784    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.591557 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.591535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lqzs\" (UniqueName: \"kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs\") pod \"payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.678558 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.678528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:41.683800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.683783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.683855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.683821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndz2s\" (UniqueName: \"kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.684654 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.684630    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.693987 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.693961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndz2s\" (UniqueName: \"kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s\") pod \"payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.760705 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.760673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:41.954064 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:41.954018    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:04:41.956274 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:41.956242    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15c0563_3ca9_4bd0_894b_508e96cdc6a6.slice/crio-49fdd151bbd15cab740baa30fd59cc84b71a9340312139c71892dd701d80338d WatchSource:0}: Error finding container 49fdd151bbd15cab740baa30fd59cc84b71a9340312139c71892dd701d80338d: Status 404 returned error can't find the container with id 49fdd151bbd15cab740baa30fd59cc84b71a9340312139c71892dd701d80338d
Sep 10 12:04:42.039619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.039592    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:04:42.041457 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:42.041433    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode498051f_6b9b_4497_bd73_448e12264583.slice/crio-53450d164f09de9a1022ebf76480ad14183aa7c4c899a29dae9f1a904b86c73e WatchSource:0}: Error finding container 53450d164f09de9a1022ebf76480ad14183aa7c4c899a29dae9f1a904b86c73e: Status 404 returned error can't find the container with id 53450d164f09de9a1022ebf76480ad14183aa7c4c899a29dae9f1a904b86c73e
Sep 10 12:04:42.564874 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.564833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" event={"ID":"e498051f-6b9b-4497-bd73-448e12264583","Type":"ContainerStarted","Data":"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"}
Sep 10 12:04:42.565318 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.564891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" event={"ID":"e498051f-6b9b-4497-bd73-448e12264583","Type":"ContainerStarted","Data":"53450d164f09de9a1022ebf76480ad14183aa7c4c899a29dae9f1a904b86c73e"}
Sep 10 12:04:42.565318 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.564930    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:42.566631 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.566605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" event={"ID":"e15c0563-3ca9-4bd0-894b-508e96cdc6a6","Type":"ContainerStarted","Data":"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"}
Sep 10 12:04:42.566725 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.566653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" event={"ID":"e15c0563-3ca9-4bd0-894b-508e96cdc6a6","Type":"ContainerStarted","Data":"49fdd151bbd15cab740baa30fd59cc84b71a9340312139c71892dd701d80338d"}
Sep 10 12:04:42.566725 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.566682    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:42.597320 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.597244    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" podStartSLOduration=1.5972226219999999 podStartE2EDuration="1.597222622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:42.593491887 +0000 UTC m=+784.173419273" watchObservedRunningTime="2026-09-10 12:04:42.597222622 +0000 UTC m=+784.177150010"
Sep 10 12:04:42.706291 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.706268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:42.722974 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.722926    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" podStartSLOduration=1.722911442 podStartE2EDuration="1.722911442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:42.620801287 +0000 UTC m=+784.200728674" watchObservedRunningTime="2026-09-10 12:04:42.722911442 +0000 UTC m=+784.302838828"
Sep 10 12:04:42.750358 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.750325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7967-139929861815728_5b50d5f7-fbba-4ae4-be53-deed51a9f5be/test-curl-7967-139929861815728/0.log"
Sep 10 12:04:42.777455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.777422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"]
Sep 10 12:04:42.779469 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.779449    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7967-139929861815728"]
Sep 10 12:04:42.795118 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.795097    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjml7\" (UniqueName: \"kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7\") pod \"5b50d5f7-fbba-4ae4-be53-deed51a9f5be\" (UID: \"5b50d5f7-fbba-4ae4-be53-deed51a9f5be\") "
Sep 10 12:04:42.797412 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.797383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7" (OuterVolumeSpecName: "kube-api-access-cjml7") pod "5b50d5f7-fbba-4ae4-be53-deed51a9f5be" (UID: "5b50d5f7-fbba-4ae4-be53-deed51a9f5be"). InnerVolumeSpecName "kube-api-access-cjml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:04:42.896256 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.896229    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjml7\" (UniqueName: \"kubernetes.io/projected/5b50d5f7-fbba-4ae4-be53-deed51a9f5be-kube-api-access-cjml7\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:04:42.950005 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:42.949973    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b50d5f7-fbba-4ae4-be53-deed51a9f5be" path="/var/lib/kubelet/pods/5b50d5f7-fbba-4ae4-be53-deed51a9f5be/volumes"
Sep 10 12:04:43.573229 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:43.573202    2573 scope.go:117] "RemoveContainer" containerID="0bddfa81549d50dbd99c48d65e48a1e931ef867e7caaaccaf5775d1a2659eff3"
Sep 10 12:04:43.573712 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:43.573278    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7967-139929861815728"
Sep 10 12:04:49.718420 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.718384    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:04:49.718884 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.718778    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b50d5f7-fbba-4ae4-be53-deed51a9f5be" containerName="test-curl-7967-139929861815728"
Sep 10 12:04:49.718884 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.718790    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b50d5f7-fbba-4ae4-be53-deed51a9f5be" containerName="test-curl-7967-139929861815728"
Sep 10 12:04:49.718884 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.718868    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b50d5f7-fbba-4ae4-be53-deed51a9f5be" containerName="test-curl-7967-139929861815728"
Sep 10 12:04:49.726571 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.726545    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:04:49.726744 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.726681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.730265 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.730244    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7a26e68a-dockercfg-bpm2c\""
Sep 10 12:04:49.730538 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.730515    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.730615 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.730550    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7a26e68a\""
Sep 10 12:04:49.736902 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.736882    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:04:49.745630 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.745601    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:04:49.760686 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.760654    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpvs\" (UniqueName: \"kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.760820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.760695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.760820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.760791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.760820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.760817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467zn\" (UniqueName: \"kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.861566 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.861525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpvs\" (UniqueName: \"kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.861566 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.861569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.861799 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.861635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.861799 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.861661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467zn\" (UniqueName: \"kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.862292 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.862270    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.862368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.862280    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:49.871487 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.871466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpvs\" (UniqueName: \"kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs\") pod \"payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") " pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:49.871639 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:49.871621    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467zn\" (UniqueName: \"kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn\") pod \"payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:50.039711 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.039620    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:50.047259 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.047236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:50.322721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.322691    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:04:50.325344 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:50.325291    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c389dd_107d_44fe_a9d0_242bdfe1afd5.slice/crio-85336bf7536ecd647c31f8d03d32e3753abeecead3dcdaddd30efea3cbf18339 WatchSource:0}: Error finding container 85336bf7536ecd647c31f8d03d32e3753abeecead3dcdaddd30efea3cbf18339: Status 404 returned error can't find the container with id 85336bf7536ecd647c31f8d03d32e3753abeecead3dcdaddd30efea3cbf18339
Sep 10 12:04:50.330522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.330497    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:04:50.331993 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:50.331970    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959bb2e3_12a8_4361_a2f5_da87829dd4fa.slice/crio-657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e WatchSource:0}: Error finding container 657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e: Status 404 returned error can't find the container with id 657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e
Sep 10 12:04:50.613005 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.612899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" event={"ID":"29c389dd-107d-44fe-a9d0-242bdfe1afd5","Type":"ContainerStarted","Data":"558bc5accdd9c412afe7fbf7c42d00533ed53f6002edd3d64b65f5be4aae7e10"}
Sep 10 12:04:50.613195 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.613017    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:04:50.613195 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.613043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" event={"ID":"29c389dd-107d-44fe-a9d0-242bdfe1afd5","Type":"ContainerStarted","Data":"85336bf7536ecd647c31f8d03d32e3753abeecead3dcdaddd30efea3cbf18339"}
Sep 10 12:04:50.614659 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.614637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" event={"ID":"959bb2e3-12a8-4361-a2f5-da87829dd4fa","Type":"ContainerStarted","Data":"8c7f774dfc5930675e8d487315cfff7b7c3163ba340e44bc4791baade29f0001"}
Sep 10 12:04:50.614747 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.614683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" event={"ID":"959bb2e3-12a8-4361-a2f5-da87829dd4fa","Type":"ContainerStarted","Data":"657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e"}
Sep 10 12:04:50.614747 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.614703    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:04:50.635750 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.635696    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" podStartSLOduration=1.6356806050000001 podStartE2EDuration="1.635680605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:50.633551815 +0000 UTC m=+792.213479201" watchObservedRunningTime="2026-09-10 12:04:50.635680605 +0000 UTC m=+792.215607995"
Sep 10 12:04:50.659290 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:50.659242    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" podStartSLOduration=1.659227557 podStartE2EDuration="1.659227557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:50.657956674 +0000 UTC m=+792.237884063" watchObservedRunningTime="2026-09-10 12:04:50.659227557 +0000 UTC m=+792.239154959"
Sep 10 12:04:52.808724 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.808685    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:04:52.812997 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.812971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.818657 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.818633    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7608bc-serving-cert\""
Sep 10 12:04:52.818836 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.818636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:04:52.818907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.818636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjsm9\""
Sep 10 12:04:52.825023 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.824918    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:04:52.884558 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.884519    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.884558 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.884555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b5x\" (UniqueName: \"kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.884820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.884693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.884820 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.884724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.985180 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.985153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b5x\" (UniqueName: \"kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.985347 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.985254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.985347 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.985270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.985479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.985347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.985685 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:04:52.985660    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-serving-cert: secret "maas-api-e2e-autores-7608bc-serving-cert" not found
Sep 10 12:04:52.985824 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:04:52.985744    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls podName:39b7b89f-e8db-47f8-8878-36ba74819189 nodeName:}" failed. No retries permitted until 2026-09-10 12:04:53.485723349 +0000 UTC m=+795.065650720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls") pod "maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" (UID: "39b7b89f-e8db-47f8-8878-36ba74819189") : secret "maas-api-e2e-autores-7608bc-serving-cert" not found
Sep 10 12:04:52.985987 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.985970    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.987943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.987921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:52.994257 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:52.994235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b5x\" (UniqueName: \"kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:53.483753 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.483716    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:04:53.488138 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.488115    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:04:53.488277 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.488259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.488646 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.488620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:53.488802 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.488665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5n6h\" (UniqueName: \"kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.488802 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.488730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.491234 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.491213    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") pod \"maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:53.491483 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.491465    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7608bc\""
Sep 10 12:04:53.491694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.491676    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7608bc-dockercfg-gvtqt\""
Sep 10 12:04:53.491769 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.491740    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.506489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.506472    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:04:53.508682 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.508659    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:04:53.575143 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.575111    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:04:53.575294 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.575167    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:04:53.589348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.589297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5n6h\" (UniqueName: \"kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.589433 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.589380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.589433 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.589416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.589523 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.589490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5h5\" (UniqueName: \"kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.590078 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.590059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.605574 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.605548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5n6h\" (UniqueName: \"kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h\") pod \"payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.690403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.690370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.690556 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.690439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5h5\" (UniqueName: \"kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.691101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.691083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.699193 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.699161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5h5\" (UniqueName: \"kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5\") pod \"payload-processing-e2e-autores-7608bc-f7989b677-49bcv\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") " pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:53.728077 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.728051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:53.810292 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.810264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:53.816423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:53.816401    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:54.002453 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.002431    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:04:54.004498 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:54.004474    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b7b89f_e8db_47f8_8878_36ba74819189.slice/crio-da663d607f0afd93f4c16cca01b92a75e53a93a6c6c10df69eab0b00eabb4449 WatchSource:0}: Error finding container da663d607f0afd93f4c16cca01b92a75e53a93a6c6c10df69eab0b00eabb4449: Status 404 returned error can't find the container with id da663d607f0afd93f4c16cca01b92a75e53a93a6c6c10df69eab0b00eabb4449
Sep 10 12:04:54.088710 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.088681    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:04:54.090275 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:54.090249    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b7b7ce_640c_4c76_abfb_8d22fd42504d.slice/crio-725e5ba676138204340dd1136f7f18e45dac69eb9c4ca0940768c9bf4b466c56 WatchSource:0}: Error finding container 725e5ba676138204340dd1136f7f18e45dac69eb9c4ca0940768c9bf4b466c56: Status 404 returned error can't find the container with id 725e5ba676138204340dd1136f7f18e45dac69eb9c4ca0940768c9bf4b466c56
Sep 10 12:04:54.099661 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.099641    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:04:54.100997 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:04:54.100974    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77796f10_6079_4982_90d0_e5193209346f.slice/crio-a8983081ea9db26a79e2c58ab663d539f551dd75fd7bb644c09ce4b345f86ce0 WatchSource:0}: Error finding container a8983081ea9db26a79e2c58ab663d539f551dd75fd7bb644c09ce4b345f86ce0: Status 404 returned error can't find the container with id a8983081ea9db26a79e2c58ab663d539f551dd75fd7bb644c09ce4b345f86ce0
Sep 10 12:04:54.638490 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.638453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" event={"ID":"d2b7b7ce-640c-4c76-abfb-8d22fd42504d","Type":"ContainerStarted","Data":"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"}
Sep 10 12:04:54.638684 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.638525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" event={"ID":"d2b7b7ce-640c-4c76-abfb-8d22fd42504d","Type":"ContainerStarted","Data":"725e5ba676138204340dd1136f7f18e45dac69eb9c4ca0940768c9bf4b466c56"}
Sep 10 12:04:54.638684 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.638595    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:04:54.639949 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.639923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" event={"ID":"39b7b89f-e8db-47f8-8878-36ba74819189","Type":"ContainerStarted","Data":"da663d607f0afd93f4c16cca01b92a75e53a93a6c6c10df69eab0b00eabb4449"}
Sep 10 12:04:54.641680 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.641657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" event={"ID":"77796f10-6079-4982-90d0-e5193209346f","Type":"ContainerStarted","Data":"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"}
Sep 10 12:04:54.641766 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.641705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" event={"ID":"77796f10-6079-4982-90d0-e5193209346f","Type":"ContainerStarted","Data":"a8983081ea9db26a79e2c58ab663d539f551dd75fd7bb644c09ce4b345f86ce0"}
Sep 10 12:04:54.641766 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.641734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:04:54.657885 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.657845    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" podStartSLOduration=1.6578314650000001 podStartE2EDuration="1.657831465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:54.655874827 +0000 UTC m=+796.235802210" watchObservedRunningTime="2026-09-10 12:04:54.657831465 +0000 UTC m=+796.237758850"
Sep 10 12:04:54.673758 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:54.673717    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" podStartSLOduration=1.6737022750000001 podStartE2EDuration="1.673702275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:54.672728388 +0000 UTC m=+796.252655777" watchObservedRunningTime="2026-09-10 12:04:54.673702275 +0000 UTC m=+796.253629661"
Sep 10 12:04:56.655956 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:56.655922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" event={"ID":"39b7b89f-e8db-47f8-8878-36ba74819189","Type":"ContainerStarted","Data":"a966f7dfbcbc4972a4d848d60992bce3c949aab5599d0f960260ab52b867cf6e"}
Sep 10 12:04:56.656367 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:56.656062    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:04:56.674176 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:04:56.674128    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" podStartSLOduration=2.597440406 podStartE2EDuration="4.674109651s" totalImagesPullingTime="2.076669245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:04:56.673388348 +0000 UTC m=+798.253315733" watchObservedRunningTime="2026-09-10 12:04:56.674109651 +0000 UTC m=+798.254037038"
Sep 10 12:05:01.621956 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:01.621884    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:05:01.622290 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:01.622076    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:05:02.666776 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:02.666746    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:05:05.652022 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:05.651988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:05:05.652462 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:05.652061    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:05:10.510472 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.510436    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:05:10.514102 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.514086    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.518500 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.518480    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-425dc7\""
Sep 10 12:05:10.518606 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.518527    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-425dc7-dockercfg-s8f6s\""
Sep 10 12:05:10.527619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.527598    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:05:10.654171 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.654134    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.654353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.654219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gkm\" (UniqueName: \"kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.754803 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.754767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.754979 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.754862    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gkm\" (UniqueName: \"kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.755464 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.755444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.763861 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.763808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gkm\" (UniqueName: \"kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm\") pod \"payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:10.824715 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:10.824687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:11.104116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:11.104088    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:05:11.106010 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:05:11.105979    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c0fb7_da99_4490_83a0_429d836506d7.slice/crio-dd14af12524d80594f75a867bd6356144fae04f691e0ebb6eb619d6355fd949d WatchSource:0}: Error finding container dd14af12524d80594f75a867bd6356144fae04f691e0ebb6eb619d6355fd949d: Status 404 returned error can't find the container with id dd14af12524d80594f75a867bd6356144fae04f691e0ebb6eb619d6355fd949d
Sep 10 12:05:11.745742 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:11.745707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" event={"ID":"654c0fb7-da99-4490-83a0-429d836506d7","Type":"ContainerStarted","Data":"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"}
Sep 10 12:05:11.746187 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:11.745799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" event={"ID":"654c0fb7-da99-4490-83a0-429d836506d7","Type":"ContainerStarted","Data":"dd14af12524d80594f75a867bd6356144fae04f691e0ebb6eb619d6355fd949d"}
Sep 10 12:05:11.746187 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:11.745835    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:11.768121 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:11.768069    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" podStartSLOduration=1.768049032 podStartE2EDuration="1.768049032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:05:11.767778583 +0000 UTC m=+813.347705970" watchObservedRunningTime="2026-09-10 12:05:11.768049032 +0000 UTC m=+813.347976422"
Sep 10 12:05:12.344969 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.344933    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:05:12.348658 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.348638    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.351387 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.351366    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-fd49b346-kserve-self-signed-certs\""
Sep 10 12:05:12.358412 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.358390    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:05:12.471257 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.471257 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.471479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.471479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.471479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471379    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-792k7\" (UniqueName: \"kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.471479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.471406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572608 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572608 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-792k7\" (UniqueName: \"kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.572855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.572770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.573056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.573000    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.573107 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.573065    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.573170 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.573150    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.575142 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.575122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.575340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.575323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.581602 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.581582    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-792k7\" (UniqueName: \"kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7\") pod \"e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") " pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.657732 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.657703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:12.937266 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:12.937194    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:05:12.940369 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:05:12.940337    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49ab529f_120f_4ca3_9897_0452a06a069b.slice/crio-493040f3817a3696651b0b665f291440f6796b567e1d840e9f70c0109d69ce48 WatchSource:0}: Error finding container 493040f3817a3696651b0b665f291440f6796b567e1d840e9f70c0109d69ce48: Status 404 returned error can't find the container with id 493040f3817a3696651b0b665f291440f6796b567e1d840e9f70c0109d69ce48
Sep 10 12:05:13.758661 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:13.758627    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" event={"ID":"49ab529f-120f-4ca3-9897-0452a06a069b","Type":"ContainerStarted","Data":"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"}
Sep 10 12:05:13.758822 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:13.758673    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" event={"ID":"49ab529f-120f-4ca3-9897-0452a06a069b","Type":"ContainerStarted","Data":"493040f3817a3696651b0b665f291440f6796b567e1d840e9f70c0109d69ce48"}
Sep 10 12:05:13.782026 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:13.781977    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" podStartSLOduration=1.781960604 podStartE2EDuration="1.781960604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:05:13.780135862 +0000 UTC m=+815.360063261" watchObservedRunningTime="2026-09-10 12:05:13.781960604 +0000 UTC m=+815.361887990"
Sep 10 12:05:22.658812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:22.658771    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:22.659372 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:22.658834    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:22.671775 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:22.671750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:22.753684 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:22.753651    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:05:22.815612 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:22.815582    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:05:29.571658 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.571621    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:05:29.572108 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.571860    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" podUID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" containerName="payload-processing" containerID="cri-o://8c7f774dfc5930675e8d487315cfff7b7c3163ba340e44bc4791baade29f0001" gracePeriod=30
Sep 10 12:05:29.580226 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.580174    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:05:29.581233 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.580968    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" podUID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" containerName="payload-pre-processing" containerID="cri-o://558bc5accdd9c412afe7fbf7c42d00533ed53f6002edd3d64b65f5be4aae7e10" gracePeriod=30
Sep 10 12:05:29.844222 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.844191    2573 generic.go:356] "Generic (PLEG): container finished" podID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" containerID="8c7f774dfc5930675e8d487315cfff7b7c3163ba340e44bc4791baade29f0001" exitCode=0
Sep 10 12:05:29.844392 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.844259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" event={"ID":"959bb2e3-12a8-4361-a2f5-da87829dd4fa","Type":"ContainerDied","Data":"8c7f774dfc5930675e8d487315cfff7b7c3163ba340e44bc4791baade29f0001"}
Sep 10 12:05:29.844392 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.844360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f" event={"ID":"959bb2e3-12a8-4361-a2f5-da87829dd4fa","Type":"ContainerDied","Data":"657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e"}
Sep 10 12:05:29.844392 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.844388    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="657fc8c45924e367523bc0b1115f1c54f69e86b828aace771df4653e6d6bd80e"
Sep 10 12:05:29.845856 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.845837    2573 generic.go:356] "Generic (PLEG): container finished" podID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" containerID="558bc5accdd9c412afe7fbf7c42d00533ed53f6002edd3d64b65f5be4aae7e10" exitCode=0
Sep 10 12:05:29.845972 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.845907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" event={"ID":"29c389dd-107d-44fe-a9d0-242bdfe1afd5","Type":"ContainerDied","Data":"558bc5accdd9c412afe7fbf7c42d00533ed53f6002edd3d64b65f5be4aae7e10"}
Sep 10 12:05:29.861090 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.861069    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:05:29.864029 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:29.864014    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:05:30.046339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046289    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-467zn\" (UniqueName: \"kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn\") pod \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") "
Sep 10 12:05:30.046339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume\") pod \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\" (UID: \"29c389dd-107d-44fe-a9d0-242bdfe1afd5\") "
Sep 10 12:05:30.046557 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046376    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tpvs\" (UniqueName: \"kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs\") pod \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") "
Sep 10 12:05:30.046557 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046396    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume\") pod \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\" (UID: \"959bb2e3-12a8-4361-a2f5-da87829dd4fa\") "
Sep 10 12:05:30.046807 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29c389dd-107d-44fe-a9d0-242bdfe1afd5" (UID: "29c389dd-107d-44fe-a9d0-242bdfe1afd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:05:30.046866 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.046780    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "959bb2e3-12a8-4361-a2f5-da87829dd4fa" (UID: "959bb2e3-12a8-4361-a2f5-da87829dd4fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:05:30.048771 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.048748    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn" (OuterVolumeSpecName: "kube-api-access-467zn") pod "29c389dd-107d-44fe-a9d0-242bdfe1afd5" (UID: "29c389dd-107d-44fe-a9d0-242bdfe1afd5"). InnerVolumeSpecName "kube-api-access-467zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:05:30.048895 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.048870    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs" (OuterVolumeSpecName: "kube-api-access-2tpvs") pod "959bb2e3-12a8-4361-a2f5-da87829dd4fa" (UID: "959bb2e3-12a8-4361-a2f5-da87829dd4fa"). InnerVolumeSpecName "kube-api-access-2tpvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:05:30.147696 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.147668    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467zn\" (UniqueName: \"kubernetes.io/projected/29c389dd-107d-44fe-a9d0-242bdfe1afd5-kube-api-access-467zn\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:05:30.147696 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.147697    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c389dd-107d-44fe-a9d0-242bdfe1afd5-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:05:30.147893 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.147706    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tpvs\" (UniqueName: \"kubernetes.io/projected/959bb2e3-12a8-4361-a2f5-da87829dd4fa-kube-api-access-2tpvs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:05:30.147893 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.147716    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959bb2e3-12a8-4361-a2f5-da87829dd4fa-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:05:30.853098 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.853055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg" event={"ID":"29c389dd-107d-44fe-a9d0-242bdfe1afd5","Type":"ContainerDied","Data":"85336bf7536ecd647c31f8d03d32e3753abeecead3dcdaddd30efea3cbf18339"}
Sep 10 12:05:30.853650 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.853140    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"
Sep 10 12:05:30.853650 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.853155    2573 scope.go:117] "RemoveContainer" containerID="558bc5accdd9c412afe7fbf7c42d00533ed53f6002edd3d64b65f5be4aae7e10"
Sep 10 12:05:30.853650 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.853074    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"
Sep 10 12:05:30.892934 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.892909    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:05:30.898774 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.898754    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7a26e68a-7bdbb6b94b-jhzsg"]
Sep 10 12:05:30.920875 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.920854    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:05:30.934031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.934000    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7a26e68a-5f764544c-cp55f"]
Sep 10 12:05:30.947693 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.947663    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" path="/var/lib/kubelet/pods/29c389dd-107d-44fe-a9d0-242bdfe1afd5/volumes"
Sep 10 12:05:30.948084 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:30.948069    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" path="/var/lib/kubelet/pods/959bb2e3-12a8-4361-a2f5-da87829dd4fa/volumes"
Sep 10 12:05:35.180752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.180716    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:05:35.181160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181147    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" containerName="payload-pre-processing"
Sep 10 12:05:35.181202 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181161    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" containerName="payload-pre-processing"
Sep 10 12:05:35.181202 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181173    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" containerName="payload-processing"
Sep 10 12:05:35.181202 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181178    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" containerName="payload-processing"
Sep 10 12:05:35.181325 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181241    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c389dd-107d-44fe-a9d0-242bdfe1afd5" containerName="payload-pre-processing"
Sep 10 12:05:35.181325 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.181248    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="959bb2e3-12a8-4361-a2f5-da87829dd4fa" containerName="payload-processing"
Sep 10 12:05:35.184366 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.184352    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.187879 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.187853    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-17efa977\""
Sep 10 12:05:35.188023 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.187936    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-17efa977-dockercfg-8m4vm\""
Sep 10 12:05:35.196721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.196695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.196925 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.196903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9md\" (UniqueName: \"kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.199063 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.199044    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:05:35.298322 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.298270    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9md\" (UniqueName: \"kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.298468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.298360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.298967 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.298949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.306787 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.306763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9md\" (UniqueName: \"kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md\") pod \"payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") " pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.494830 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.494756    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.769500 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.769478    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:05:35.771166 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:05:35.771131    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b51198_40e1_48a9_8050_3c1381d3e05e.slice/crio-ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000 WatchSource:0}: Error finding container ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000: Status 404 returned error can't find the container with id ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000
Sep 10 12:05:35.882800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.882768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" event={"ID":"43b51198-40e1-48a9-8050-3c1381d3e05e","Type":"ContainerStarted","Data":"9518d1a3870400ba2b4b1a85b754aef4bdbdb336739408c5c98a8bcb82edada6"}
Sep 10 12:05:35.882947 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.882826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" event={"ID":"43b51198-40e1-48a9-8050-3c1381d3e05e","Type":"ContainerStarted","Data":"ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000"}
Sep 10 12:05:35.882947 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.882878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:05:35.907503 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:35.907445    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" podStartSLOduration=0.907424195 podStartE2EDuration="907.424195ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:05:35.903931074 +0000 UTC m=+837.483858461" watchObservedRunningTime="2026-09-10 12:05:35.907424195 +0000 UTC m=+837.487351586"
Sep 10 12:05:46.890185 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:05:46.890153    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:06:05.775742 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:05.775705    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:06:05.776403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:05.775946    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" podUID="43b51198-40e1-48a9-8050-3c1381d3e05e" containerName="payload-processing" containerID="cri-o://9518d1a3870400ba2b4b1a85b754aef4bdbdb336739408c5c98a8bcb82edada6" gracePeriod=30
Sep 10 12:06:06.047727 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.047697    2573 generic.go:356] "Generic (PLEG): container finished" podID="43b51198-40e1-48a9-8050-3c1381d3e05e" containerID="9518d1a3870400ba2b4b1a85b754aef4bdbdb336739408c5c98a8bcb82edada6" exitCode=0
Sep 10 12:06:06.047888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.047781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" event={"ID":"43b51198-40e1-48a9-8050-3c1381d3e05e","Type":"ContainerDied","Data":"9518d1a3870400ba2b4b1a85b754aef4bdbdb336739408c5c98a8bcb82edada6"}
Sep 10 12:06:06.047961 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.047893    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n" event={"ID":"43b51198-40e1-48a9-8050-3c1381d3e05e","Type":"ContainerDied","Data":"ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000"}
Sep 10 12:06:06.047961 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.047920    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5c8efbe9d44b0a79f360af59b103511eb38af81265f51d7247c55760c0d000"
Sep 10 12:06:06.050209 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.050189    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:06:06.113152 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.113126    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t9md\" (UniqueName: \"kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md\") pod \"43b51198-40e1-48a9-8050-3c1381d3e05e\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") "
Sep 10 12:06:06.113331 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.113230    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume\") pod \"43b51198-40e1-48a9-8050-3c1381d3e05e\" (UID: \"43b51198-40e1-48a9-8050-3c1381d3e05e\") "
Sep 10 12:06:06.113651 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.113623    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43b51198-40e1-48a9-8050-3c1381d3e05e" (UID: "43b51198-40e1-48a9-8050-3c1381d3e05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:06:06.115522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.115500    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md" (OuterVolumeSpecName: "kube-api-access-6t9md") pod "43b51198-40e1-48a9-8050-3c1381d3e05e" (UID: "43b51198-40e1-48a9-8050-3c1381d3e05e"). InnerVolumeSpecName "kube-api-access-6t9md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:06:06.214723 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.214693    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9md\" (UniqueName: \"kubernetes.io/projected/43b51198-40e1-48a9-8050-3c1381d3e05e-kube-api-access-6t9md\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:06:06.214723 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:06.214719    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b51198-40e1-48a9-8050-3c1381d3e05e-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:06:07.053439 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:07.053413    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"
Sep 10 12:06:07.070126 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:07.070095    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:06:07.076643 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:07.076620    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-17efa977-54cf7864d6-thx5n"]
Sep 10 12:06:08.948472 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:08.948444    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43b51198-40e1-48a9-8050-3c1381d3e05e" path="/var/lib/kubelet/pods/43b51198-40e1-48a9-8050-3c1381d3e05e/volumes"
Sep 10 12:06:11.082008 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.081968    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:11.082641 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.082618    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43b51198-40e1-48a9-8050-3c1381d3e05e" containerName="payload-processing"
Sep 10 12:06:11.082738 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.082642    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="43b51198-40e1-48a9-8050-3c1381d3e05e" containerName="payload-processing"
Sep 10 12:06:11.082795 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.082787    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="43b51198-40e1-48a9-8050-3c1381d3e05e" containerName="payload-processing"
Sep 10 12:06:11.086129 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.086107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.091365 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.091339    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eb1d9122\""
Sep 10 12:06:11.091511 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.091464    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eb1d9122-dockercfg-ng8zh\""
Sep 10 12:06:11.101334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.101296    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:11.161088 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.161058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.161265 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.161111    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9kn\" (UniqueName: \"kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.261686 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.261644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.261864 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.261698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9kn\" (UniqueName: \"kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.262364 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.262345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.274055 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.274029    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9kn\" (UniqueName: \"kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn\") pod \"payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") " pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.396503 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.396462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:11.672248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:11.672217    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:11.674722 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:06:11.674697    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec77b80e_fa97_44f0_814e_326a4c37e83f.slice/crio-4393c80e53f17796fa569097267954533fa4b52b29acc3f2789ba0a1021d134f WatchSource:0}: Error finding container 4393c80e53f17796fa569097267954533fa4b52b29acc3f2789ba0a1021d134f: Status 404 returned error can't find the container with id 4393c80e53f17796fa569097267954533fa4b52b29acc3f2789ba0a1021d134f
Sep 10 12:06:12.082276 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:12.082180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" event={"ID":"ec77b80e-fa97-44f0-814e-326a4c37e83f","Type":"ContainerStarted","Data":"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"}
Sep 10 12:06:12.082276 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:12.082259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" event={"ID":"ec77b80e-fa97-44f0-814e-326a4c37e83f","Type":"ContainerStarted","Data":"4393c80e53f17796fa569097267954533fa4b52b29acc3f2789ba0a1021d134f"}
Sep 10 12:06:12.082724 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:12.082330    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:12.117404 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:12.117355    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" podStartSLOduration=1.117338533 podStartE2EDuration="1.117338533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:06:12.116008349 +0000 UTC m=+873.695935736" watchObservedRunningTime="2026-09-10 12:06:12.117338533 +0000 UTC m=+873.697265919"
Sep 10 12:06:23.090038 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:23.090010    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:38.930405 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:38.930325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:06:38.931176 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:38.931155    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:06:38.953993 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:38.953970    2573 scope.go:117] "RemoveContainer" containerID="35551bac1f8d8bf481843042a5c096babb59118202a263af87c715bf01399abc"
Sep 10 12:06:41.365975 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.365938    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:41.366435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.366197    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" podUID="ec77b80e-fa97-44f0-814e-326a4c37e83f" containerName="payload-processing" containerID="cri-o://7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977" gracePeriod=30
Sep 10 12:06:41.619072 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.619016    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:41.749402 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.749370    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume\") pod \"ec77b80e-fa97-44f0-814e-326a4c37e83f\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") "
Sep 10 12:06:41.749567 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.749412    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc9kn\" (UniqueName: \"kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn\") pod \"ec77b80e-fa97-44f0-814e-326a4c37e83f\" (UID: \"ec77b80e-fa97-44f0-814e-326a4c37e83f\") "
Sep 10 12:06:41.749810 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.749788    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec77b80e-fa97-44f0-814e-326a4c37e83f" (UID: "ec77b80e-fa97-44f0-814e-326a4c37e83f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:06:41.751722 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.751700    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn" (OuterVolumeSpecName: "kube-api-access-mc9kn") pod "ec77b80e-fa97-44f0-814e-326a4c37e83f" (UID: "ec77b80e-fa97-44f0-814e-326a4c37e83f"). InnerVolumeSpecName "kube-api-access-mc9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:06:41.850296 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.850261    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec77b80e-fa97-44f0-814e-326a4c37e83f-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:06:41.850296 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:41.850288    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9kn\" (UniqueName: \"kubernetes.io/projected/ec77b80e-fa97-44f0-814e-326a4c37e83f-kube-api-access-mc9kn\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:06:42.238547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.238514    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec77b80e-fa97-44f0-814e-326a4c37e83f" containerID="7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977" exitCode=0
Sep 10 12:06:42.238739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.238564    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"
Sep 10 12:06:42.238739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.238599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" event={"ID":"ec77b80e-fa97-44f0-814e-326a4c37e83f","Type":"ContainerDied","Data":"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"}
Sep 10 12:06:42.238739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.238655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c" event={"ID":"ec77b80e-fa97-44f0-814e-326a4c37e83f","Type":"ContainerDied","Data":"4393c80e53f17796fa569097267954533fa4b52b29acc3f2789ba0a1021d134f"}
Sep 10 12:06:42.238739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.238677    2573 scope.go:117] "RemoveContainer" containerID="7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"
Sep 10 12:06:42.248099 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.248082    2573 scope.go:117] "RemoveContainer" containerID="7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"
Sep 10 12:06:42.248419 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:06:42.248402    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977\": container with ID starting with 7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977 not found: ID does not exist" containerID="7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"
Sep 10 12:06:42.248492 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.248430    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977"} err="failed to get container status \"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977\": rpc error: code = NotFound desc = could not find container \"7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977\": container with ID starting with 7bec127cf857b303504fa442c0d3b44aa5c204c07490c3a13cc4d5a7acd73977 not found: ID does not exist"
Sep 10 12:06:42.259058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.259033    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:42.262999 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.262978    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb1d9122-85c8489d75-bcq5c"]
Sep 10 12:06:42.947792 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:42.947756    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec77b80e-fa97-44f0-814e-326a4c37e83f" path="/var/lib/kubelet/pods/ec77b80e-fa97-44f0-814e-326a4c37e83f/volumes"
Sep 10 12:06:44.255815 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.255196    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:06:44.256281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.255824    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec77b80e-fa97-44f0-814e-326a4c37e83f" containerName="payload-processing"
Sep 10 12:06:44.256281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.255840    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec77b80e-fa97-44f0-814e-326a4c37e83f" containerName="payload-processing"
Sep 10 12:06:44.256281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.255955    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec77b80e-fa97-44f0-814e-326a4c37e83f" containerName="payload-processing"
Sep 10 12:06:44.260987 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.260967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.265666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.265642    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7608bc\"/\"openshift-service-ca.crt\""
Sep 10 12:06:44.266386 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.265985    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7608bc\"/\"kube-root-ca.crt\""
Sep 10 12:06:44.266386 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.266107    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7608bc\"/\"default-dockercfg-8f4sl\""
Sep 10 12:06:44.266386 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.266201    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7608bc\"/\"autores-model-7608bc-kserve-self-signed-certs\""
Sep 10 12:06:44.270196 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.270173    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:06:44.270549 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.270528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.270759 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.270740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.270908 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.270894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.271017 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.271006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.272685 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.272666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.272894 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.272876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5px\" (UniqueName: \"kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374209 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5px\" (UniqueName: \"kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374681 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374614    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374681 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.374775 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.374688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.376694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.376665    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.376862 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.376844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.384936 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.384912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5px\" (UniqueName: \"kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px\") pod \"autores-model-7608bc-kserve-7897d6b499-xlzgt\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") " pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.575456 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.575363    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:44.850930 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:44.850906    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:06:44.853144 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:06:44.853117    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf58405d0_afb7_4d09_880d_668421eb71ea.slice/crio-c8d39aea755fd935207593ede2d82f76286832cdf6390f104e0843650c671832 WatchSource:0}: Error finding container c8d39aea755fd935207593ede2d82f76286832cdf6390f104e0843650c671832: Status 404 returned error can't find the container with id c8d39aea755fd935207593ede2d82f76286832cdf6390f104e0843650c671832
Sep 10 12:06:45.256203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:45.256171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" event={"ID":"f58405d0-afb7-4d09-880d-668421eb71ea","Type":"ContainerStarted","Data":"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"}
Sep 10 12:06:45.256203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:45.256209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" event={"ID":"f58405d0-afb7-4d09-880d-668421eb71ea","Type":"ContainerStarted","Data":"c8d39aea755fd935207593ede2d82f76286832cdf6390f104e0843650c671832"}
Sep 10 12:06:45.274046 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:45.273997    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" podStartSLOduration=1.273980368 podStartE2EDuration="1.273980368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:06:45.272923754 +0000 UTC m=+906.852851133" watchObservedRunningTime="2026-09-10 12:06:45.273980368 +0000 UTC m=+906.853907804"
Sep 10 12:06:48.644600 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.644564    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:06:48.648509 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.648486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.651294 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.651268    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4d5fbdf0-dockercfg-5zd2s\""
Sep 10 12:06:48.652886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.652868    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4d5fbdf0\""
Sep 10 12:06:48.663915 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.663891    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:06:48.713698 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.713663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lstp\" (UniqueName: \"kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.713853 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.713735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.815144 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.815109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lstp\" (UniqueName: \"kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.815331 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.815163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.815858 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.815839    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.828267 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.828247    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lstp\" (UniqueName: \"kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp\") pod \"payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") " pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:48.960243 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:48.960166    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:49.234835 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:49.234782    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:06:49.237500 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:06:49.237474    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10da40bf_bf13_4687_b5ca_3c54e2bbca79.slice/crio-ffa4ce32ea4f529fdfff38846fa817613e7933a76758033c63b42882e0bb0cb0 WatchSource:0}: Error finding container ffa4ce32ea4f529fdfff38846fa817613e7933a76758033c63b42882e0bb0cb0: Status 404 returned error can't find the container with id ffa4ce32ea4f529fdfff38846fa817613e7933a76758033c63b42882e0bb0cb0
Sep 10 12:06:49.279667 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:49.279642    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" event={"ID":"10da40bf-bf13-4687-b5ca-3c54e2bbca79","Type":"ContainerStarted","Data":"ffa4ce32ea4f529fdfff38846fa817613e7933a76758033c63b42882e0bb0cb0"}
Sep 10 12:06:50.286910 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:50.286871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" event={"ID":"10da40bf-bf13-4687-b5ca-3c54e2bbca79","Type":"ContainerStarted","Data":"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"}
Sep 10 12:06:50.287382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:50.286946    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:06:50.305766 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:50.305714    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" podStartSLOduration=2.305697964 podStartE2EDuration="2.305697964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:06:50.304650365 +0000 UTC m=+911.884577752" watchObservedRunningTime="2026-09-10 12:06:50.305697964 +0000 UTC m=+911.885625350"
Sep 10 12:06:54.576117 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:54.576083    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:54.576117 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:54.576124    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:54.588519 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:54.588495    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:06:55.325111 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:06:55.325083    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:07:01.295294 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:01.295258    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:07:18.722059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:18.722021    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:07:18.724379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:18.722327    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" podUID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" containerName="payload-processing" containerID="cri-o://d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2" gracePeriod=30
Sep 10 12:07:18.980671 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:18.980616    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:07:19.120191 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.120157    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume\") pod \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") "
Sep 10 12:07:19.120369 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.120238    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lstp\" (UniqueName: \"kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp\") pod \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\" (UID: \"10da40bf-bf13-4687-b5ca-3c54e2bbca79\") "
Sep 10 12:07:19.120585 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.120562    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10da40bf-bf13-4687-b5ca-3c54e2bbca79" (UID: "10da40bf-bf13-4687-b5ca-3c54e2bbca79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:07:19.122463 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.122440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp" (OuterVolumeSpecName: "kube-api-access-7lstp") pod "10da40bf-bf13-4687-b5ca-3c54e2bbca79" (UID: "10da40bf-bf13-4687-b5ca-3c54e2bbca79"). InnerVolumeSpecName "kube-api-access-7lstp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:19.221516 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.221473    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10da40bf-bf13-4687-b5ca-3c54e2bbca79-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:19.221516 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.221512    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lstp\" (UniqueName: \"kubernetes.io/projected/10da40bf-bf13-4687-b5ca-3c54e2bbca79-kube-api-access-7lstp\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:19.441980 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.441942    2573 generic.go:356] "Generic (PLEG): container finished" podID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" containerID="d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2" exitCode=0
Sep 10 12:07:19.442157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.442005    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"
Sep 10 12:07:19.442157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.442027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" event={"ID":"10da40bf-bf13-4687-b5ca-3c54e2bbca79","Type":"ContainerDied","Data":"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"}
Sep 10 12:07:19.442157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.442104    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9" event={"ID":"10da40bf-bf13-4687-b5ca-3c54e2bbca79","Type":"ContainerDied","Data":"ffa4ce32ea4f529fdfff38846fa817613e7933a76758033c63b42882e0bb0cb0"}
Sep 10 12:07:19.442157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.442128    2573 scope.go:117] "RemoveContainer" containerID="d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"
Sep 10 12:07:19.451559 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.451542    2573 scope.go:117] "RemoveContainer" containerID="d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"
Sep 10 12:07:19.451821 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:07:19.451801    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2\": container with ID starting with d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2 not found: ID does not exist" containerID="d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"
Sep 10 12:07:19.451874 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.451830    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2"} err="failed to get container status \"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2\": rpc error: code = NotFound desc = could not find container \"d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2\": container with ID starting with d7c835ef56ac9e350ea48f197f5c05ddcf71037afab5de31303af0ee688eaed2 not found: ID does not exist"
Sep 10 12:07:19.466288 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.466266    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:07:19.472323 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:19.472286    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4d5fbdf0-5cd478975c-jrfg9"]
Sep 10 12:07:20.946814 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:20.946784    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" path="/var/lib/kubelet/pods/10da40bf-bf13-4687-b5ca-3c54e2bbca79/volumes"
Sep 10 12:07:26.257245 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.257209    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:07:26.257728 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.257492    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" podUID="39b7b89f-e8db-47f8-8878-36ba74819189" containerName="maas-api" containerID="cri-o://a966f7dfbcbc4972a4d848d60992bce3c949aab5599d0f960260ab52b867cf6e" gracePeriod=30
Sep 10 12:07:26.390785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.390753    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:07:26.391034 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.390985    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" podUID="77796f10-6079-4982-90d0-e5193209346f" containerName="payload-processing" containerID="cri-o://05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb" gracePeriod=30
Sep 10 12:07:26.399425 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.398616    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:07:26.399425 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.398822    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" podUID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" containerName="payload-pre-processing" containerID="cri-o://37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e" gracePeriod=30
Sep 10 12:07:26.482463 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.482415    2573 generic.go:356] "Generic (PLEG): container finished" podID="39b7b89f-e8db-47f8-8878-36ba74819189" containerID="a966f7dfbcbc4972a4d848d60992bce3c949aab5599d0f960260ab52b867cf6e" exitCode=0
Sep 10 12:07:26.482603 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.482470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" event={"ID":"39b7b89f-e8db-47f8-8878-36ba74819189","Type":"ContainerDied","Data":"a966f7dfbcbc4972a4d848d60992bce3c949aab5599d0f960260ab52b867cf6e"}
Sep 10 12:07:26.523474 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.523450    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:07:26.587934 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.587903    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs\") pod \"39b7b89f-e8db-47f8-8878-36ba74819189\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") "
Sep 10 12:07:26.588093 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.587985    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca\") pod \"39b7b89f-e8db-47f8-8878-36ba74819189\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") "
Sep 10 12:07:26.588093 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.588021    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5b5x\" (UniqueName: \"kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x\") pod \"39b7b89f-e8db-47f8-8878-36ba74819189\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") "
Sep 10 12:07:26.588612 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.588581    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39b7b89f-e8db-47f8-8878-36ba74819189" (UID: "39b7b89f-e8db-47f8-8878-36ba74819189"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:07:26.591593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.591565    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39b7b89f-e8db-47f8-8878-36ba74819189" (UID: "39b7b89f-e8db-47f8-8878-36ba74819189"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:07:26.591804 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.591783    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x" (OuterVolumeSpecName: "kube-api-access-k5b5x") pod "39b7b89f-e8db-47f8-8878-36ba74819189" (UID: "39b7b89f-e8db-47f8-8878-36ba74819189"). InnerVolumeSpecName "kube-api-access-k5b5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:26.660703 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.660671    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:07:26.669482 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.669460    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:07:26.688792 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.688756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume\") pod \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") "
Sep 10 12:07:26.688929 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.688810    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") pod \"39b7b89f-e8db-47f8-8878-36ba74819189\" (UID: \"39b7b89f-e8db-47f8-8878-36ba74819189\") "
Sep 10 12:07:26.688929 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.688827    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg5h5\" (UniqueName: \"kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5\") pod \"77796f10-6079-4982-90d0-e5193209346f\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") "
Sep 10 12:07:26.688929 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.688846    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume\") pod \"77796f10-6079-4982-90d0-e5193209346f\" (UID: \"77796f10-6079-4982-90d0-e5193209346f\") "
Sep 10 12:07:26.688929 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.688864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5n6h\" (UniqueName: \"kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h\") pod \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\" (UID: \"d2b7b7ce-640c-4c76-abfb-8d22fd42504d\") "
Sep 10 12:07:26.689182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.689163    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.689244 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.689170    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2b7b7ce-640c-4c76-abfb-8d22fd42504d" (UID: "d2b7b7ce-640c-4c76-abfb-8d22fd42504d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:07:26.689244 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.689185    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b7b89f-e8db-47f8-8878-36ba74819189-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.689244 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.689206    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b5x\" (UniqueName: \"kubernetes.io/projected/39b7b89f-e8db-47f8-8878-36ba74819189-kube-api-access-k5b5x\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.689429 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.689244    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77796f10-6079-4982-90d0-e5193209346f" (UID: "77796f10-6079-4982-90d0-e5193209346f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:07:26.691089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.691070    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5" (OuterVolumeSpecName: "kube-api-access-fg5h5") pod "77796f10-6079-4982-90d0-e5193209346f" (UID: "77796f10-6079-4982-90d0-e5193209346f"). InnerVolumeSpecName "kube-api-access-fg5h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:26.691175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.691106    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h" (OuterVolumeSpecName: "kube-api-access-j5n6h") pod "d2b7b7ce-640c-4c76-abfb-8d22fd42504d" (UID: "d2b7b7ce-640c-4c76-abfb-8d22fd42504d"). InnerVolumeSpecName "kube-api-access-j5n6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:26.691175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.691110    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39b7b89f-e8db-47f8-8878-36ba74819189" (UID: "39b7b89f-e8db-47f8-8878-36ba74819189"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:07:26.790435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.790337    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.790435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.790369    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b7b89f-e8db-47f8-8878-36ba74819189-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.790435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.790378    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5h5\" (UniqueName: \"kubernetes.io/projected/77796f10-6079-4982-90d0-e5193209346f-kube-api-access-fg5h5\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.790435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.790388    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77796f10-6079-4982-90d0-e5193209346f-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:26.790435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:26.790398    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5n6h\" (UniqueName: \"kubernetes.io/projected/d2b7b7ce-640c-4c76-abfb-8d22fd42504d-kube-api-access-j5n6h\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:27.489804 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.489774    2573 generic.go:356] "Generic (PLEG): container finished" podID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" containerID="37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e" exitCode=0
Sep 10 12:07:27.490233 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.489820    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"
Sep 10 12:07:27.490233 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.489865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" event={"ID":"d2b7b7ce-640c-4c76-abfb-8d22fd42504d","Type":"ContainerDied","Data":"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"}
Sep 10 12:07:27.490233 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.489930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6" event={"ID":"d2b7b7ce-640c-4c76-abfb-8d22fd42504d","Type":"ContainerDied","Data":"725e5ba676138204340dd1136f7f18e45dac69eb9c4ca0940768c9bf4b466c56"}
Sep 10 12:07:27.490233 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.489968    2573 scope.go:117] "RemoveContainer" containerID="37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"
Sep 10 12:07:27.491522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.491464    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"
Sep 10 12:07:27.491522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.491492    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t" event={"ID":"39b7b89f-e8db-47f8-8878-36ba74819189","Type":"ContainerDied","Data":"da663d607f0afd93f4c16cca01b92a75e53a93a6c6c10df69eab0b00eabb4449"}
Sep 10 12:07:27.493058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.493012    2573 generic.go:356] "Generic (PLEG): container finished" podID="77796f10-6079-4982-90d0-e5193209346f" containerID="05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb" exitCode=0
Sep 10 12:07:27.493163 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.493065    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"
Sep 10 12:07:27.493163 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.493095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" event={"ID":"77796f10-6079-4982-90d0-e5193209346f","Type":"ContainerDied","Data":"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"}
Sep 10 12:07:27.493163 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.493141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv" event={"ID":"77796f10-6079-4982-90d0-e5193209346f","Type":"ContainerDied","Data":"a8983081ea9db26a79e2c58ab663d539f551dd75fd7bb644c09ce4b345f86ce0"}
Sep 10 12:07:27.499889 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.499869    2573 scope.go:117] "RemoveContainer" containerID="37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"
Sep 10 12:07:27.500160 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:07:27.500141    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e\": container with ID starting with 37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e not found: ID does not exist" containerID="37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"
Sep 10 12:07:27.500234 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.500166    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e"} err="failed to get container status \"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e\": rpc error: code = NotFound desc = could not find container \"37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e\": container with ID starting with 37820f9c62620ce186367747bdf2a91d0aa866f3f4cb5eac17028e1f5f129c9e not found: ID does not exist"
Sep 10 12:07:27.500234 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.500182    2573 scope.go:117] "RemoveContainer" containerID="a966f7dfbcbc4972a4d848d60992bce3c949aab5599d0f960260ab52b867cf6e"
Sep 10 12:07:27.508318 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.508286    2573 scope.go:117] "RemoveContainer" containerID="05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"
Sep 10 12:07:27.516942 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.516925    2573 scope.go:117] "RemoveContainer" containerID="05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"
Sep 10 12:07:27.517216 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:07:27.517193    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb\": container with ID starting with 05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb not found: ID does not exist" containerID="05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"
Sep 10 12:07:27.517273 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.517226    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb"} err="failed to get container status \"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb\": rpc error: code = NotFound desc = could not find container \"05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb\": container with ID starting with 05aac066eb559a1a17fde6c45a116844e5e25ef61770d692b4806e43abdefecb not found: ID does not exist"
Sep 10 12:07:27.517273 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.517261    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:07:27.522257 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.522240    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7608bc-7c494bcc66-dtq7t"]
Sep 10 12:07:27.532168 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.532116    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:07:27.538654 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.538636    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7608bc-5f659b65c8-6zfd6"]
Sep 10 12:07:27.550362 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.550342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:07:27.555384 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:27.555367    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7608bc-f7989b677-49bcv"]
Sep 10 12:07:28.955393 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:28.955361    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b7b89f-e8db-47f8-8878-36ba74819189" path="/var/lib/kubelet/pods/39b7b89f-e8db-47f8-8878-36ba74819189/volumes"
Sep 10 12:07:28.955797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:28.955716    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77796f10-6079-4982-90d0-e5193209346f" path="/var/lib/kubelet/pods/77796f10-6079-4982-90d0-e5193209346f/volumes"
Sep 10 12:07:28.956037 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:28.956024    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" path="/var/lib/kubelet/pods/d2b7b7ce-640c-4c76-abfb-8d22fd42504d/volumes"
Sep 10 12:07:31.922733 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:31.922697    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:07:32.954943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.954907    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:07:32.955562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955537    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" containerName="payload-pre-processing"
Sep 10 12:07:32.955562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955560    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" containerName="payload-pre-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955573    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77796f10-6079-4982-90d0-e5193209346f" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955578    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77796f10-6079-4982-90d0-e5193209346f" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955587    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b7b89f-e8db-47f8-8878-36ba74819189" containerName="maas-api"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955593    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b7b89f-e8db-47f8-8878-36ba74819189" containerName="maas-api"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955610    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955618    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955692    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="77796f10-6079-4982-90d0-e5193209346f" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955702    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="10da40bf-bf13-4687-b5ca-3c54e2bbca79" containerName="payload-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955712    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b7b7ce-640c-4c76-abfb-8d22fd42504d" containerName="payload-pre-processing"
Sep 10 12:07:32.955921 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.955717    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b7b89f-e8db-47f8-8878-36ba74819189" containerName="maas-api"
Sep 10 12:07:32.960125 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.960103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:32.964100 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.964081    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89d91a\"/\"kube-root-ca.crt\""
Sep 10 12:07:32.964100 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.964093    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89d91a\"/\"test-model-89d91a-kserve-self-signed-certs\""
Sep 10 12:07:32.964268 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.964084    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-89d91a\"/\"default-dockercfg-wxvpk\""
Sep 10 12:07:32.964268 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.964085    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-89d91a\"/\"openshift-service-ca.crt\""
Sep 10 12:07:32.969390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:32.969368    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:07:33.045600 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.045600 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtjr\" (UniqueName: \"kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.045797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.045797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.045797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.045896 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.045843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.146628 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.146808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.146808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.146808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.146808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtjr\" (UniqueName: \"kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.147033 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.146883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.147221 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.147176    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.147336 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.147225    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.147608 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.147588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.149222 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.149196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.149378 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.149363    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.154882 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.154863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtjr\" (UniqueName: \"kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr\") pod \"test-model-89d91a-kserve-7f8c58f48f-4l47r\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") " pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.270919 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.270835    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:33.546841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:33.546747    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:07:33.555064 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:07:33.555037    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4738f853_11bc_4263_9406_9efdfa1edc8a.slice/crio-c95265e95e9eb048cbc4587cff646e13febe6ba4dfe23fb0077c0ffe5a4ac421 WatchSource:0}: Error finding container c95265e95e9eb048cbc4587cff646e13febe6ba4dfe23fb0077c0ffe5a4ac421: Status 404 returned error can't find the container with id c95265e95e9eb048cbc4587cff646e13febe6ba4dfe23fb0077c0ffe5a4ac421
Sep 10 12:07:34.536190 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:34.536155    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" event={"ID":"4738f853-11bc-4263-9406-9efdfa1edc8a","Type":"ContainerStarted","Data":"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"}
Sep 10 12:07:34.536190 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:34.536201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" event={"ID":"4738f853-11bc-4263-9406-9efdfa1edc8a","Type":"ContainerStarted","Data":"c95265e95e9eb048cbc4587cff646e13febe6ba4dfe23fb0077c0ffe5a4ac421"}
Sep 10 12:07:34.555886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:34.555811    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" podStartSLOduration=2.555796072 podStartE2EDuration="2.555796072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:07:34.55486437 +0000 UTC m=+956.134791755" watchObservedRunningTime="2026-09-10 12:07:34.555796072 +0000 UTC m=+956.135723459"
Sep 10 12:07:35.529809 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:35.529768    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:07:43.271548 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:43.271512    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:43.271964 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:43.271567    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:43.284802 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:43.284776    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:43.597416 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:43.597334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:07:46.943117 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:46.943076    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" podUID="f58405d0-afb7-4d09-880d-668421eb71ea" containerName="main" containerID="cri-o://52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71" gracePeriod=45
Sep 10 12:07:47.196694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.196643    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:07:47.276549 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276516    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276707 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276610    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j5px\" (UniqueName: \"kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276707 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276707 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276772    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir\") pod \"f58405d0-afb7-4d09-880d-668421eb71ea\" (UID: \"f58405d0-afb7-4d09-880d-668421eb71ea\") "
Sep 10 12:07:47.276999 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276969    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home" (OuterVolumeSpecName: "home") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:47.276999 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.276961    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:47.277197 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.277177    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:47.277266 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.277186    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.277266 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.277219    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.279080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.279060    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm" (OuterVolumeSpecName: "dshm") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:47.279189 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.279129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px" (OuterVolumeSpecName: "kube-api-access-9j5px") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "kube-api-access-9j5px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:47.279235 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.279202    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f58405d0-afb7-4d09-880d-668421eb71ea" (UID: "f58405d0-afb7-4d09-880d-668421eb71ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:07:47.378536 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.378509    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58405d0-afb7-4d09-880d-668421eb71ea-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.378536 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.378533    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.378718 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.378541    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58405d0-afb7-4d09-880d-668421eb71ea-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.378718 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.378551    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5px\" (UniqueName: \"kubernetes.io/projected/f58405d0-afb7-4d09-880d-668421eb71ea-kube-api-access-9j5px\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:47.609352 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.609228    2573 generic.go:356] "Generic (PLEG): container finished" podID="f58405d0-afb7-4d09-880d-668421eb71ea" containerID="52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71" exitCode=0
Sep 10 12:07:47.609352 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.609274    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"
Sep 10 12:07:47.609352 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.609326    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" event={"ID":"f58405d0-afb7-4d09-880d-668421eb71ea","Type":"ContainerDied","Data":"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"}
Sep 10 12:07:47.609648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.609376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt" event={"ID":"f58405d0-afb7-4d09-880d-668421eb71ea","Type":"ContainerDied","Data":"c8d39aea755fd935207593ede2d82f76286832cdf6390f104e0843650c671832"}
Sep 10 12:07:47.609648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.609398    2573 scope.go:117] "RemoveContainer" containerID="52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"
Sep 10 12:07:47.619434 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.619413    2573 scope.go:117] "RemoveContainer" containerID="52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"
Sep 10 12:07:47.619713 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:07:47.619694    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71\": container with ID starting with 52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71 not found: ID does not exist" containerID="52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"
Sep 10 12:07:47.619797 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.619720    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71"} err="failed to get container status \"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71\": rpc error: code = NotFound desc = could not find container \"52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71\": container with ID starting with 52f0f89738eb5844596ecb5881a9d4edb7033cd1a51523449f22568afe2e7c71 not found: ID does not exist"
Sep 10 12:07:47.638190 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.638163    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:07:47.641915 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:47.641893    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7608bc/autores-model-7608bc-kserve-7897d6b499-xlzgt"]
Sep 10 12:07:48.947025 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:48.946993    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f58405d0-afb7-4d09-880d-668421eb71ea" path="/var/lib/kubelet/pods/f58405d0-afb7-4d09-880d-668421eb71ea/volumes"
Sep 10 12:07:50.555181 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.555141    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" podUID="49ab529f-120f-4ca3-9897-0452a06a069b" containerName="main" containerID="cri-o://0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0" gracePeriod=45
Sep 10 12:07:50.807508 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.807452    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:07:50.915019 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.914993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915029    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915067    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-792k7\" (UniqueName: \"kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915115    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915155    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir\") pod \"49ab529f-120f-4ca3-9897-0452a06a069b\" (UID: \"49ab529f-120f-4ca3-9897-0452a06a069b\") "
Sep 10 12:07:50.915484 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915323    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home" (OuterVolumeSpecName: "home") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:50.915484 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915422    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache" (OuterVolumeSpecName: "model-cache") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:50.915593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:50.915593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915515    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:50.915593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.915532    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:50.917467 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.917445    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm" (OuterVolumeSpecName: "dshm") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:07:50.917553 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.917487    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:07:50.917599 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:50.917579    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7" (OuterVolumeSpecName: "kube-api-access-792k7") pod "49ab529f-120f-4ca3-9897-0452a06a069b" (UID: "49ab529f-120f-4ca3-9897-0452a06a069b"). InnerVolumeSpecName "kube-api-access-792k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:07:51.016886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.016854    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:51.016886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.016882    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49ab529f-120f-4ca3-9897-0452a06a069b-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:51.016886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.016892    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-792k7\" (UniqueName: \"kubernetes.io/projected/49ab529f-120f-4ca3-9897-0452a06a069b-kube-api-access-792k7\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:51.017082 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.016901    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49ab529f-120f-4ca3-9897-0452a06a069b-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:07:51.632348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.632282    2573 generic.go:356] "Generic (PLEG): container finished" podID="49ab529f-120f-4ca3-9897-0452a06a069b" containerID="0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0" exitCode=0
Sep 10 12:07:51.632781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.632365    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"
Sep 10 12:07:51.632781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.632370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" event={"ID":"49ab529f-120f-4ca3-9897-0452a06a069b","Type":"ContainerDied","Data":"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"}
Sep 10 12:07:51.632781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.632412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg" event={"ID":"49ab529f-120f-4ca3-9897-0452a06a069b","Type":"ContainerDied","Data":"493040f3817a3696651b0b665f291440f6796b567e1d840e9f70c0109d69ce48"}
Sep 10 12:07:51.632781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.632431    2573 scope.go:117] "RemoveContainer" containerID="0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"
Sep 10 12:07:51.641132 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.641118    2573 scope.go:117] "RemoveContainer" containerID="0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"
Sep 10 12:07:51.641534 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:07:51.641515    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0\": container with ID starting with 0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0 not found: ID does not exist" containerID="0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"
Sep 10 12:07:51.641588 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.641543    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0"} err="failed to get container status \"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0\": rpc error: code = NotFound desc = could not find container \"0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0\": container with ID starting with 0295857946e210348508d3bf3a5cc1a8ff8641fbd68a563499afebbb63220af0 not found: ID does not exist"
Sep 10 12:07:51.652197 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.652176    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:07:51.657427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:51.657408    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-fd49b346-kserve-74584d9465-dpxwg"]
Sep 10 12:07:52.947048 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:07:52.947017    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ab529f-120f-4ca3-9897-0452a06a069b" path="/var/lib/kubelet/pods/49ab529f-120f-4ca3-9897-0452a06a069b/volumes"
Sep 10 12:08:09.170534 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.170455    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:08:09.171038 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171017    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58405d0-afb7-4d09-880d-668421eb71ea" containerName="main"
Sep 10 12:08:09.171038 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171040    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58405d0-afb7-4d09-880d-668421eb71ea" containerName="main"
Sep 10 12:08:09.171181 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171086    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ab529f-120f-4ca3-9897-0452a06a069b" containerName="main"
Sep 10 12:08:09.171181 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171096    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ab529f-120f-4ca3-9897-0452a06a069b" containerName="main"
Sep 10 12:08:09.171181 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ab529f-120f-4ca3-9897-0452a06a069b" containerName="main"
Sep 10 12:08:09.171181 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.171179    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58405d0-afb7-4d09-880d-668421eb71ea" containerName="main"
Sep 10 12:08:09.174560 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.174544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.183403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.179045    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-425dc7\"/\"openshift-service-ca.crt\""
Sep 10 12:08:09.183403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.179219    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-425dc7\"/\"default-dockercfg-dwklf\""
Sep 10 12:08:09.183403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.179352    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-425dc7\"/\"test-model-425dc7-kserve-self-signed-certs\""
Sep 10 12:08:09.183403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.179045    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-425dc7\"/\"kube-root-ca.crt\""
Sep 10 12:08:09.185637 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.185318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:08:09.193933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.193908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.194069 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.194025    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.194204 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.194146    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.194358 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.194338    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.194434 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.194394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmkl7\" (UniqueName: \"kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.194489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.194453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295866 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295995 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295995 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295995 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmkl7\" (UniqueName: \"kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295995 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.295995 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.295976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.296348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.296286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.296484 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.296349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.296484 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.296386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.298378 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.298359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.298627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.298609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.307318 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.307269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmkl7\" (UniqueName: \"kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7\") pod \"test-model-425dc7-kserve-87dcd567f-fsh8g\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") " pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.489348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.489256    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:09.769255 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:09.769227    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:08:09.771322 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:08:09.771272    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07f0d548_5d1e_4959_b8cb_b16ac937e437.slice/crio-2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c WatchSource:0}: Error finding container 2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c: Status 404 returned error can't find the container with id 2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c
Sep 10 12:08:10.729548 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:10.729510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" event={"ID":"07f0d548-5d1e-4959-b8cb-b16ac937e437","Type":"ContainerStarted","Data":"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"}
Sep 10 12:08:10.729908 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:10.729562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" event={"ID":"07f0d548-5d1e-4959-b8cb-b16ac937e437","Type":"ContainerStarted","Data":"2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c"}
Sep 10 12:08:10.750454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:10.750404    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" podStartSLOduration=1.750387396 podStartE2EDuration="1.750387396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:08:10.749127237 +0000 UTC m=+992.329054625" watchObservedRunningTime="2026-09-10 12:08:10.750387396 +0000 UTC m=+992.330314816"
Sep 10 12:08:14.651344 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.651292    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:08:14.655088 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.655065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.658902 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.658878    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-316319-serving-cert\""
Sep 10 12:08:14.659031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.659008    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjsm9\""
Sep 10 12:08:14.659164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.659146    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:08:14.673204 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.673184    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:08:14.747462 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.747429    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.747627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.747473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.747627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.747531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.747627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.747571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl5w\" (UniqueName: \"kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.848907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.848872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.849072 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.848923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.849072 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.848956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.849072 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.848988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl5w\" (UniqueName: \"kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.849726 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.849696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.851605 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.851588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.851663 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.851648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.858856 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.858832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl5w\" (UniqueName: \"kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w\") pod \"maas-api-e2e-explicit-316319-59fdc787dd-fzrfc\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:14.966274 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:14.966200    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:15.248198 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:15.248170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:08:15.249713 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:08:15.249686    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4981359c_3dbc_4b46_be00_2e81183a1164.slice/crio-3ce14e7fdd5c633b389783d402a2fb8573fa0b947fb9de1e77b16b127c9a3913 WatchSource:0}: Error finding container 3ce14e7fdd5c633b389783d402a2fb8573fa0b947fb9de1e77b16b127c9a3913: Status 404 returned error can't find the container with id 3ce14e7fdd5c633b389783d402a2fb8573fa0b947fb9de1e77b16b127c9a3913
Sep 10 12:08:15.251061 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:15.251043    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:08:15.765342 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:15.765293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" event={"ID":"4981359c-3dbc-4b46-be00-2e81183a1164","Type":"ContainerStarted","Data":"3ce14e7fdd5c633b389783d402a2fb8573fa0b947fb9de1e77b16b127c9a3913"}
Sep 10 12:08:16.773276 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:16.773239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" event={"ID":"4981359c-3dbc-4b46-be00-2e81183a1164","Type":"ContainerStarted","Data":"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"}
Sep 10 12:08:16.773702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:16.773321    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:16.796787 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:16.796737    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" podStartSLOduration=2.260655996 podStartE2EDuration="2.796715503s" totalImagesPullingTime="536.059507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:08:16.796206372 +0000 UTC m=+998.376133760" watchObservedRunningTime="2026-09-10 12:08:16.796715503 +0000 UTC m=+998.376642888"
Sep 10 12:08:19.490063 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:19.490026    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:19.490474 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:19.490118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:19.502816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:19.502792    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:19.807289 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:19.807208    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:08:22.785116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:22.785082    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:08:45.353204 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.353171    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:08:45.357944 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.357923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.362079 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.362058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-316319\"/\"default-dockercfg-rc57x\""
Sep 10 12:08:45.362219 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.362061    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-316319\"/\"openshift-service-ca.crt\""
Sep 10 12:08:45.362219 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.362086    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-316319\"/\"explicit-model-316319-kserve-self-signed-certs\""
Sep 10 12:08:45.362219 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.362187    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-316319\"/\"kube-root-ca.crt\""
Sep 10 12:08:45.366857 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.366837    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:08:45.456062 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.456062 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqsq\" (UniqueName: \"kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.456272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.456272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.456382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.456382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.456337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557133 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557133 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqsq\" (UniqueName: \"kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557670 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557675    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.557721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.557675    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.559666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.559642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.559814 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.559795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.565829 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.565811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqsq\" (UniqueName: \"kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq\") pod \"explicit-model-316319-kserve-7d45bb9755-rzfcn\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") " pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.667496 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.667461    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:45.941663 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:45.941640    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:08:45.943725 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:08:45.943692    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3993dbb_db9d_4a98_bb37_4d8f82565f20.slice/crio-b3187b9c39f25c3294864e9c43308dad5bf77b69a6525af477ed1d1a0334b1f7 WatchSource:0}: Error finding container b3187b9c39f25c3294864e9c43308dad5bf77b69a6525af477ed1d1a0334b1f7: Status 404 returned error can't find the container with id b3187b9c39f25c3294864e9c43308dad5bf77b69a6525af477ed1d1a0334b1f7
Sep 10 12:08:46.938094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:46.938052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" event={"ID":"e3993dbb-db9d-4a98-bb37-4d8f82565f20","Type":"ContainerStarted","Data":"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"}
Sep 10 12:08:46.938094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:46.938096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" event={"ID":"e3993dbb-db9d-4a98-bb37-4d8f82565f20","Type":"ContainerStarted","Data":"b3187b9c39f25c3294864e9c43308dad5bf77b69a6525af477ed1d1a0334b1f7"}
Sep 10 12:08:46.958202 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:46.958150    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" podStartSLOduration=1.958135007 podStartE2EDuration="1.958135007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:08:46.955851883 +0000 UTC m=+1028.535779271" watchObservedRunningTime="2026-09-10 12:08:46.958135007 +0000 UTC m=+1028.538062435"
Sep 10 12:08:55.668173 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:55.668134    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:55.668694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:55.668194    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:55.680893 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:55.680867    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:55.992092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:55.992006    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:08:56.103468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:08:56.103435    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:08:57.070896 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:57.070866    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:08:57.071272 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:57.070943    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:08:57.570925901 +0000 UTC m=+1039.150853266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:08:57.576951 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:57.576918    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:08:57.577131 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:57.576982    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:08:58.576968841 +0000 UTC m=+1040.156896204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:08:58.587348 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:58.587289    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:08:58.587733 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:08:58.587384    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:00.587368245 +0000 UTC m=+1042.167295610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:00.607379 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:00.607342    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:00.607757 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:00.607423    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:04.607408526 +0000 UTC m=+1046.187335894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:02.816115 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.816083    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:02.820784 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.820769    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:02.823907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.823878    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0f6353c1-dockercfg-gclfw\""
Sep 10 12:09:02.823907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.823882    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0f6353c1\""
Sep 10 12:09:02.833424 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.833400    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:02.929934 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.929903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lcc\" (UniqueName: \"kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:02.930090 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:02.929940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.031000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.030966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lcc\" (UniqueName: \"kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.031000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.031005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.031793 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.031766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.041745 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.041720    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lcc\" (UniqueName: \"kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc\") pod \"payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") " pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.131587 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.131500    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:03.411099 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:03.411073    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:03.412604 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:09:03.412578    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259d0dfb_d353_4deb_9c02_7d8b0af39422.slice/crio-83b5e3a45a04f946e7de720e620ed9fab7040fce23b97c2a25cfddca2dccbb26 WatchSource:0}: Error finding container 83b5e3a45a04f946e7de720e620ed9fab7040fce23b97c2a25cfddca2dccbb26: Status 404 returned error can't find the container with id 83b5e3a45a04f946e7de720e620ed9fab7040fce23b97c2a25cfddca2dccbb26
Sep 10 12:09:04.031336 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:04.031282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" event={"ID":"259d0dfb-d353-4deb-9c02-7d8b0af39422","Type":"ContainerStarted","Data":"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"}
Sep 10 12:09:04.031728 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:04.031389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" event={"ID":"259d0dfb-d353-4deb-9c02-7d8b0af39422","Type":"ContainerStarted","Data":"83b5e3a45a04f946e7de720e620ed9fab7040fce23b97c2a25cfddca2dccbb26"}
Sep 10 12:09:04.031728 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:04.031419    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:04.052662 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:04.052604    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" podStartSLOduration=2.052585024 podStartE2EDuration="2.052585024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:04.051107792 +0000 UTC m=+1045.631035184" watchObservedRunningTime="2026-09-10 12:09:04.052585024 +0000 UTC m=+1045.632512413"
Sep 10 12:09:04.644082 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:04.644041    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:04.644249 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:04.644126    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:12.644102694 +0000 UTC m=+1054.224030059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:07.081505 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.081470    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:09:07.081968 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.081851    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" podUID="4981359c-3dbc-4b46-be00-2e81183a1164" containerName="maas-api" containerID="cri-o://5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a" gracePeriod=30
Sep 10 12:09:07.331168 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.331144    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:09:07.472374 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.472341    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca\") pod \"4981359c-3dbc-4b46-be00-2e81183a1164\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") "
Sep 10 12:09:07.472541 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.472441    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kl5w\" (UniqueName: \"kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w\") pod \"4981359c-3dbc-4b46-be00-2e81183a1164\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") "
Sep 10 12:09:07.472541 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.472486    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls\") pod \"4981359c-3dbc-4b46-be00-2e81183a1164\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") "
Sep 10 12:09:07.472541 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.472517    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs\") pod \"4981359c-3dbc-4b46-be00-2e81183a1164\" (UID: \"4981359c-3dbc-4b46-be00-2e81183a1164\") "
Sep 10 12:09:07.472763 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.472725    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4981359c-3dbc-4b46-be00-2e81183a1164" (UID: "4981359c-3dbc-4b46-be00-2e81183a1164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:07.473030 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.473005    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4981359c-3dbc-4b46-be00-2e81183a1164-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:07.474785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.474755    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w" (OuterVolumeSpecName: "kube-api-access-7kl5w") pod "4981359c-3dbc-4b46-be00-2e81183a1164" (UID: "4981359c-3dbc-4b46-be00-2e81183a1164"). InnerVolumeSpecName "kube-api-access-7kl5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:07.475160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.475140    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4981359c-3dbc-4b46-be00-2e81183a1164" (UID: "4981359c-3dbc-4b46-be00-2e81183a1164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:09:07.475238 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.475158    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4981359c-3dbc-4b46-be00-2e81183a1164" (UID: "4981359c-3dbc-4b46-be00-2e81183a1164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:09:07.574220 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.574190    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kl5w\" (UniqueName: \"kubernetes.io/projected/4981359c-3dbc-4b46-be00-2e81183a1164-kube-api-access-7kl5w\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:07.574220 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.574217    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:07.574220 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:07.574227    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4981359c-3dbc-4b46-be00-2e81183a1164-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:08.052992 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.052959    2573 generic.go:356] "Generic (PLEG): container finished" podID="4981359c-3dbc-4b46-be00-2e81183a1164" containerID="5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a" exitCode=0
Sep 10 12:09:08.053194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.053008    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"
Sep 10 12:09:08.053194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.053042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" event={"ID":"4981359c-3dbc-4b46-be00-2e81183a1164","Type":"ContainerDied","Data":"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"}
Sep 10 12:09:08.053194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.053094    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc" event={"ID":"4981359c-3dbc-4b46-be00-2e81183a1164","Type":"ContainerDied","Data":"3ce14e7fdd5c633b389783d402a2fb8573fa0b947fb9de1e77b16b127c9a3913"}
Sep 10 12:09:08.053194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.053116    2573 scope.go:117] "RemoveContainer" containerID="5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"
Sep 10 12:09:08.063095 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.063082    2573 scope.go:117] "RemoveContainer" containerID="5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"
Sep 10 12:09:08.063395 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:08.063375    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a\": container with ID starting with 5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a not found: ID does not exist" containerID="5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"
Sep 10 12:09:08.063501 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.063401    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a"} err="failed to get container status \"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a\": rpc error: code = NotFound desc = could not find container \"5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a\": container with ID starting with 5a9caaaa418268d00e83c67f0bfaba81a2612d9ff0c80d9532fd8a7ed82bb86a not found: ID does not exist"
Sep 10 12:09:08.080533 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.080495    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:09:08.085900 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.085874    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-316319-59fdc787dd-fzrfc"]
Sep 10 12:09:08.947872 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:08.947843    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4981359c-3dbc-4b46-be00-2e81183a1164" path="/var/lib/kubelet/pods/4981359c-3dbc-4b46-be00-2e81183a1164/volumes"
Sep 10 12:09:12.725185 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:12.725146    2573 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-self-signed-certs: secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:12.725612 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:12.725223    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs podName:e3993dbb-db9d-4a98-bb37-4d8f82565f20 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:28.725205326 +0000 UTC m=+1070.305132689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs") pod "explicit-model-316319-kserve-7d45bb9755-rzfcn" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20") : secret "explicit-model-316319-kserve-self-signed-certs" not found
Sep 10 12:09:13.011740 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.011646    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" podUID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" containerName="main" containerID="cri-o://7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc" gracePeriod=45
Sep 10 12:09:13.269253 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.269191    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:09:13.330712 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330685    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.330858 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330733    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.330858 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.330858 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330783    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.330997 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqqsq\" (UniqueName: \"kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.330997 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330921    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs\") pod \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\" (UID: \"e3993dbb-db9d-4a98-bb37-4d8f82565f20\") "
Sep 10 12:09:13.331089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.330994    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:13.331089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.331074    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home" (OuterVolumeSpecName: "home") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:13.331201 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.331089    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:13.331323 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.331281    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:13.331384 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.331329    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:13.331384 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.331341    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:13.333116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.333095    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm" (OuterVolumeSpecName: "dshm") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:13.333230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.333129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:09:13.333230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.333145    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq" (OuterVolumeSpecName: "kube-api-access-vqqsq") pod "e3993dbb-db9d-4a98-bb37-4d8f82565f20" (UID: "e3993dbb-db9d-4a98-bb37-4d8f82565f20"). InnerVolumeSpecName "kube-api-access-vqqsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:13.432081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.432039    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3993dbb-db9d-4a98-bb37-4d8f82565f20-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:13.432081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.432073    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3993dbb-db9d-4a98-bb37-4d8f82565f20-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:13.432081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:13.432086    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqsq\" (UniqueName: \"kubernetes.io/projected/e3993dbb-db9d-4a98-bb37-4d8f82565f20-kube-api-access-vqqsq\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:14.090339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.090283    2573 generic.go:356] "Generic (PLEG): container finished" podID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" containerID="7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc" exitCode=0
Sep 10 12:09:14.090339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.090349    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"
Sep 10 12:09:14.090829 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.090376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" event={"ID":"e3993dbb-db9d-4a98-bb37-4d8f82565f20","Type":"ContainerDied","Data":"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"}
Sep 10 12:09:14.090829 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.090424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn" event={"ID":"e3993dbb-db9d-4a98-bb37-4d8f82565f20","Type":"ContainerDied","Data":"b3187b9c39f25c3294864e9c43308dad5bf77b69a6525af477ed1d1a0334b1f7"}
Sep 10 12:09:14.090829 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.090441    2573 scope.go:117] "RemoveContainer" containerID="7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"
Sep 10 12:09:14.100020 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.099999    2573 scope.go:117] "RemoveContainer" containerID="7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"
Sep 10 12:09:14.100351 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:14.100331    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc\": container with ID starting with 7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc not found: ID does not exist" containerID="7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"
Sep 10 12:09:14.100426 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.100363    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc"} err="failed to get container status \"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc\": rpc error: code = NotFound desc = could not find container \"7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc\": container with ID starting with 7894d1d32d9436d594b84034e334713e1aac852d8baaa065e67b348e272577bc not found: ID does not exist"
Sep 10 12:09:14.118876 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.118848    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:09:14.126503 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.126475    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-316319/explicit-model-316319-kserve-7d45bb9755-rzfcn"]
Sep 10 12:09:14.947607 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:14.947575    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" path="/var/lib/kubelet/pods/e3993dbb-db9d-4a98-bb37-4d8f82565f20/volumes"
Sep 10 12:09:15.039028 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:15.038999    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:15.656479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:15.656447    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:09:16.814440 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:16.814410    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:09:27.730228 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:27.730191    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:09:27.731379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:27.731323    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" podUID="e498051f-6b9b-4497-bd73-448e12264583" containerName="payload-processing" containerID="cri-o://05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5" gracePeriod=30
Sep 10 12:09:27.734262 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:27.734234    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:09:27.734526 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:27.734485    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" podUID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" containerName="payload-pre-processing" containerID="cri-o://7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665" gracePeriod=30
Sep 10 12:09:28.012203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.012178    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:09:28.014996 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.014977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:09:28.076553 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.076522    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume\") pod \"e498051f-6b9b-4497-bd73-448e12264583\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") "
Sep 10 12:09:28.076736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.076607    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lqzs\" (UniqueName: \"kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs\") pod \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") "
Sep 10 12:09:28.076736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.076648    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume\") pod \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\" (UID: \"e15c0563-3ca9-4bd0-894b-508e96cdc6a6\") "
Sep 10 12:09:28.076736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.076673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndz2s\" (UniqueName: \"kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s\") pod \"e498051f-6b9b-4497-bd73-448e12264583\" (UID: \"e498051f-6b9b-4497-bd73-448e12264583\") "
Sep 10 12:09:28.076968 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.076944    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e498051f-6b9b-4497-bd73-448e12264583" (UID: "e498051f-6b9b-4497-bd73-448e12264583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:28.077064 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.077037    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e15c0563-3ca9-4bd0-894b-508e96cdc6a6" (UID: "e15c0563-3ca9-4bd0-894b-508e96cdc6a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:28.078973 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.078947    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs" (OuterVolumeSpecName: "kube-api-access-6lqzs") pod "e15c0563-3ca9-4bd0-894b-508e96cdc6a6" (UID: "e15c0563-3ca9-4bd0-894b-508e96cdc6a6"). InnerVolumeSpecName "kube-api-access-6lqzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:28.079090 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.079003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s" (OuterVolumeSpecName: "kube-api-access-ndz2s") pod "e498051f-6b9b-4497-bd73-448e12264583" (UID: "e498051f-6b9b-4497-bd73-448e12264583"). InnerVolumeSpecName "kube-api-access-ndz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:28.165056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.165021    2573 generic.go:356] "Generic (PLEG): container finished" podID="e498051f-6b9b-4497-bd73-448e12264583" containerID="05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5" exitCode=0
Sep 10 12:09:28.165248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.165064    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"
Sep 10 12:09:28.165248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.165112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" event={"ID":"e498051f-6b9b-4497-bd73-448e12264583","Type":"ContainerDied","Data":"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"}
Sep 10 12:09:28.165248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.165174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw" event={"ID":"e498051f-6b9b-4497-bd73-448e12264583","Type":"ContainerDied","Data":"53450d164f09de9a1022ebf76480ad14183aa7c4c899a29dae9f1a904b86c73e"}
Sep 10 12:09:28.165248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.165202    2573 scope.go:117] "RemoveContainer" containerID="05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"
Sep 10 12:09:28.166598 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.166576    2573 generic.go:356] "Generic (PLEG): container finished" podID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" containerID="7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665" exitCode=0
Sep 10 12:09:28.166702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.166622    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"
Sep 10 12:09:28.166702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.166640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" event={"ID":"e15c0563-3ca9-4bd0-894b-508e96cdc6a6","Type":"ContainerDied","Data":"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"}
Sep 10 12:09:28.166702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.166673    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq" event={"ID":"e15c0563-3ca9-4bd0-894b-508e96cdc6a6","Type":"ContainerDied","Data":"49fdd151bbd15cab740baa30fd59cc84b71a9340312139c71892dd701d80338d"}
Sep 10 12:09:28.175828 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.175811    2573 scope.go:117] "RemoveContainer" containerID="05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"
Sep 10 12:09:28.176123 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:28.176101    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5\": container with ID starting with 05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5 not found: ID does not exist" containerID="05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"
Sep 10 12:09:28.176223 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.176127    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5"} err="failed to get container status \"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5\": rpc error: code = NotFound desc = could not find container \"05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5\": container with ID starting with 05d18c4d4110fa184d589f0a6ea45dbf0779d9eae1f9db3e74e357040b88d9d5 not found: ID does not exist"
Sep 10 12:09:28.176223 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.176144    2573 scope.go:117] "RemoveContainer" containerID="7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"
Sep 10 12:09:28.177749 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.177684    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lqzs\" (UniqueName: \"kubernetes.io/projected/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-kube-api-access-6lqzs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:28.177749 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.177705    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15c0563-3ca9-4bd0-894b-508e96cdc6a6-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:28.177749 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.177715    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndz2s\" (UniqueName: \"kubernetes.io/projected/e498051f-6b9b-4497-bd73-448e12264583-kube-api-access-ndz2s\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:28.177749 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.177725    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e498051f-6b9b-4497-bd73-448e12264583-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:28.184965 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.184948    2573 scope.go:117] "RemoveContainer" containerID="7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"
Sep 10 12:09:28.185241 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:28.185223    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665\": container with ID starting with 7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665 not found: ID does not exist" containerID="7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"
Sep 10 12:09:28.185322 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.185249    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665"} err="failed to get container status \"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665\": rpc error: code = NotFound desc = could not find container \"7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665\": container with ID starting with 7103630f30e7e518b8f4465fbe66542a713968588cac24bfc1cd40a57da81665 not found: ID does not exist"
Sep 10 12:09:28.188931 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.188909    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:09:28.193839 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.193817    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89d91a-85d9f5487c-zdtbw"]
Sep 10 12:09:28.206708 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.206684    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:09:28.211950 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.211930    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89d91a-6b4466b694-thjcq"]
Sep 10 12:09:28.947006 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.946972    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" path="/var/lib/kubelet/pods/e15c0563-3ca9-4bd0-894b-508e96cdc6a6/volumes"
Sep 10 12:09:28.947400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:28.947349    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e498051f-6b9b-4497-bd73-448e12264583" path="/var/lib/kubelet/pods/e498051f-6b9b-4497-bd73-448e12264583/volumes"
Sep 10 12:09:30.676485 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:30.676394    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" podUID="4738f853-11bc-4263-9406-9efdfa1edc8a" containerName="main" containerID="cri-o://e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214" gracePeriod=45
Sep 10 12:09:30.922403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:30.922376    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:09:31.004845 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.004757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.004877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.004904    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.004958    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005158 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005005    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005158 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005031    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:31.005158 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005038    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjtjr\" (UniqueName: \"kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr\") pod \"4738f853-11bc-4263-9406-9efdfa1edc8a\" (UID: \"4738f853-11bc-4263-9406-9efdfa1edc8a\") "
Sep 10 12:09:31.005348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005170    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:31.005348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005278    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home" (OuterVolumeSpecName: "home") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:31.005652 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005629    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.005789 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005775    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.005912 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.005879    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.007417 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.007398    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm" (OuterVolumeSpecName: "dshm") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:31.007826 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.007809    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr" (OuterVolumeSpecName: "kube-api-access-jjtjr") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "kube-api-access-jjtjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:31.007826 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.007816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4738f853-11bc-4263-9406-9efdfa1edc8a" (UID: "4738f853-11bc-4263-9406-9efdfa1edc8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:09:31.106709 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.106666    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4738f853-11bc-4263-9406-9efdfa1edc8a-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.106709 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.106701    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtjr\" (UniqueName: \"kubernetes.io/projected/4738f853-11bc-4263-9406-9efdfa1edc8a-kube-api-access-jjtjr\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.106709 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.106711    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4738f853-11bc-4263-9406-9efdfa1edc8a-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:31.186344 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.186276    2573 generic.go:356] "Generic (PLEG): container finished" podID="4738f853-11bc-4263-9406-9efdfa1edc8a" containerID="e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214" exitCode=0
Sep 10 12:09:31.186344 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.186338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" event={"ID":"4738f853-11bc-4263-9406-9efdfa1edc8a","Type":"ContainerDied","Data":"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"}
Sep 10 12:09:31.186576 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.186361    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"
Sep 10 12:09:31.186576 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.186387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r" event={"ID":"4738f853-11bc-4263-9406-9efdfa1edc8a","Type":"ContainerDied","Data":"c95265e95e9eb048cbc4587cff646e13febe6ba4dfe23fb0077c0ffe5a4ac421"}
Sep 10 12:09:31.186576 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.186406    2573 scope.go:117] "RemoveContainer" containerID="e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"
Sep 10 12:09:31.195899 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.195876    2573 scope.go:117] "RemoveContainer" containerID="e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"
Sep 10 12:09:31.196199 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:31.196179    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214\": container with ID starting with e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214 not found: ID does not exist" containerID="e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"
Sep 10 12:09:31.196263 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.196209    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214"} err="failed to get container status \"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214\": rpc error: code = NotFound desc = could not find container \"e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214\": container with ID starting with e52a84c5200e458685087d7736f002cf54a88d5524629a72aa4b59ee49b4a214 not found: ID does not exist"
Sep 10 12:09:31.208280 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.208252    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:09:31.216415 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.216389    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-89d91a/test-model-89d91a-kserve-7f8c58f48f-4l47r"]
Sep 10 12:09:31.833624 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:31.833588    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" podUID="07f0d548-5d1e-4959-b8cb-b16ac937e437" containerName="main" containerID="cri-o://33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8" gracePeriod=45
Sep 10 12:09:32.095846 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.095785    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:09:32.194495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.194462    2573 generic.go:356] "Generic (PLEG): container finished" podID="07f0d548-5d1e-4959-b8cb-b16ac937e437" containerID="33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8" exitCode=0
Sep 10 12:09:32.194678 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.194508    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"
Sep 10 12:09:32.194678 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.194522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" event={"ID":"07f0d548-5d1e-4959-b8cb-b16ac937e437","Type":"ContainerDied","Data":"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"}
Sep 10 12:09:32.194678 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.194555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g" event={"ID":"07f0d548-5d1e-4959-b8cb-b16ac937e437","Type":"ContainerDied","Data":"2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c"}
Sep 10 12:09:32.194678 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.194573    2573 scope.go:117] "RemoveContainer" containerID="33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"
Sep 10 12:09:32.203375 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.203353    2573 scope.go:117] "RemoveContainer" containerID="33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"
Sep 10 12:09:32.203703 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:32.203683    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8\": container with ID starting with 33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8 not found: ID does not exist" containerID="33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"
Sep 10 12:09:32.203762 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.203712    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8"} err="failed to get container status \"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8\": rpc error: code = NotFound desc = could not find container \"33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8\": container with ID starting with 33c412e47db1c63145bd782ce8d03d2f3516263c0492fb9e7bdefcd0238b0da8 not found: ID does not exist"
Sep 10 12:09:32.217046 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217015    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217268 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217186    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217347 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217281    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmkl7\" (UniqueName: \"kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217410 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217347    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir\") pod \"07f0d548-5d1e-4959-b8cb-b16ac937e437\" (UID: \"07f0d548-5d1e-4959-b8cb-b16ac937e437\") "
Sep 10 12:09:32.217461 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217422    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home" (OuterVolumeSpecName: "home") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:32.217519 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217478    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache" (OuterVolumeSpecName: "model-cache") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:32.217670 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217651    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:32.217744 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217709    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.217744 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.217727    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.219601 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.219580    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7" (OuterVolumeSpecName: "kube-api-access-gmkl7") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "kube-api-access-gmkl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:32.219751 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.219723    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm" (OuterVolumeSpecName: "dshm") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:09:32.219751 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.219723    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07f0d548-5d1e-4959-b8cb-b16ac937e437" (UID: "07f0d548-5d1e-4959-b8cb-b16ac937e437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:09:32.319080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.319040    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f0d548-5d1e-4959-b8cb-b16ac937e437-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.319080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.319072    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.319080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.319084    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmkl7\" (UniqueName: \"kubernetes.io/projected/07f0d548-5d1e-4959-b8cb-b16ac937e437-kube-api-access-gmkl7\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.319352 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.319094    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f0d548-5d1e-4959-b8cb-b16ac937e437-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:32.517029 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.516999    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:09:32.517487 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:32.517466    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07f0d548_5d1e_4959_b8cb_b16ac937e437.slice/crio-2f1729f27b26a67ea447899c32f720020d5f06bbcc757fb7751c4cf427bc635c\": RecentStats: unable to find data in memory cache]"
Sep 10 12:09:32.519976 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.519951    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-425dc7/test-model-425dc7-kserve-87dcd567f-fsh8g"]
Sep 10 12:09:32.948390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.948358    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f0d548-5d1e-4959-b8cb-b16ac937e437" path="/var/lib/kubelet/pods/07f0d548-5d1e-4959-b8cb-b16ac937e437/volumes"
Sep 10 12:09:32.948776 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:32.948756    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4738f853-11bc-4263-9406-9efdfa1edc8a" path="/var/lib/kubelet/pods/4738f853-11bc-4263-9406-9efdfa1edc8a/volumes"
Sep 10 12:09:33.596850 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.596817    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"]
Sep 10 12:09:33.597220 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597205    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" containerName="payload-pre-processing"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597225    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" containerName="payload-pre-processing"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597241    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4981359c-3dbc-4b46-be00-2e81183a1164" containerName="maas-api"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597248    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4981359c-3dbc-4b46-be00-2e81183a1164" containerName="maas-api"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597262    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" containerName="main"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" containerName="main"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597274    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4738f853-11bc-4263-9406-9efdfa1edc8a" containerName="main"
Sep 10 12:09:33.597281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597280    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4738f853-11bc-4263-9406-9efdfa1edc8a" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597292    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e498051f-6b9b-4497-bd73-448e12264583" containerName="payload-processing"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597310    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e498051f-6b9b-4497-bd73-448e12264583" containerName="payload-processing"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597322    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f0d548-5d1e-4959-b8cb-b16ac937e437" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597327    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f0d548-5d1e-4959-b8cb-b16ac937e437" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597381    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f0d548-5d1e-4959-b8cb-b16ac937e437" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597391    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4738f853-11bc-4263-9406-9efdfa1edc8a" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597398    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e498051f-6b9b-4497-bd73-448e12264583" containerName="payload-processing"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597406    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3993dbb-db9d-4a98-bb37-4d8f82565f20" containerName="main"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597411    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4981359c-3dbc-4b46-be00-2e81183a1164" containerName="maas-api"
Sep 10 12:09:33.597618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.597426    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e15c0563-3ca9-4bd0-894b-508e96cdc6a6" containerName="payload-pre-processing"
Sep 10 12:09:33.601966 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.601949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:33.604877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.604848    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j7ls2\""
Sep 10 12:09:33.609116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.609093    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"]
Sep 10 12:09:33.731172 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.731136    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l\" (UID: \"7d592e33-9bd1-43de-b16e-74ef1c8e0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:33.831989 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.831953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l\" (UID: \"7d592e33-9bd1-43de-b16e-74ef1c8e0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:33.832652 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.832628    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l\" (UID: \"7d592e33-9bd1-43de-b16e-74ef1c8e0084\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:33.913816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:33.913786    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:34.187624 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:34.187433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"]
Sep 10 12:09:34.190548 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:09:34.190518    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d592e33_9bd1_43de_b16e_74ef1c8e0084.slice/crio-5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b WatchSource:0}: Error finding container 5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b: Status 404 returned error can't find the container with id 5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b
Sep 10 12:09:34.208084 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:34.208059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l" event={"ID":"7d592e33-9bd1-43de-b16e-74ef1c8e0084","Type":"ContainerStarted","Data":"5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b"}
Sep 10 12:09:37.228232 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:37.228205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l" event={"ID":"7d592e33-9bd1-43de-b16e-74ef1c8e0084","Type":"ContainerStarted","Data":"f98f350e14e9967075a0c8d03885b5ed93df6c83cf2d504d002d22abe521904a"}
Sep 10 12:09:38.235548 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.235511    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d592e33-9bd1-43de-b16e-74ef1c8e0084" containerID="f98f350e14e9967075a0c8d03885b5ed93df6c83cf2d504d002d22abe521904a" exitCode=0
Sep 10 12:09:38.235984 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.235591    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l" event={"ID":"7d592e33-9bd1-43de-b16e-74ef1c8e0084","Type":"ContainerDied","Data":"f98f350e14e9967075a0c8d03885b5ed93df6c83cf2d504d002d22abe521904a"}
Sep 10 12:09:38.363685 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.363661    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:38.474869 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.474841    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca\") pod \"7d592e33-9bd1-43de-b16e-74ef1c8e0084\" (UID: \"7d592e33-9bd1-43de-b16e-74ef1c8e0084\") "
Sep 10 12:09:38.475250 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.475227    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d592e33-9bd1-43de-b16e-74ef1c8e0084" (UID: "7d592e33-9bd1-43de-b16e-74ef1c8e0084"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:38.576243 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:38.576132    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d592e33-9bd1-43de-b16e-74ef1c8e0084-maas-api-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:39.240786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:39.240690    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"
Sep 10 12:09:39.240786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:39.240689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l" event={"ID":"7d592e33-9bd1-43de-b16e-74ef1c8e0084","Type":"ContainerDied","Data":"5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b"}
Sep 10 12:09:39.241252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:39.240813    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0c9adf78095b7983acd6643170cd2d61edbca5c6ef82f0447ec3d0fedf366b"
Sep 10 12:09:40.867147 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.867116    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:09:40.867570 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.867557    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d592e33-9bd1-43de-b16e-74ef1c8e0084" containerName="revoke-keys"
Sep 10 12:09:40.867570 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.867572    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d592e33-9bd1-43de-b16e-74ef1c8e0084" containerName="revoke-keys"
Sep 10 12:09:40.867649 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.867630    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d592e33-9bd1-43de-b16e-74ef1c8e0084" containerName="revoke-keys"
Sep 10 12:09:40.872408 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.872384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.875510 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.875488    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c8a24e\"/\"default-dockercfg-kk56v\""
Sep 10 12:09:40.875794 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.875775    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c8a24e\"/\"openshift-service-ca.crt\""
Sep 10 12:09:40.876621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.876605    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c8a24e\"/\"kube-root-ca.crt\""
Sep 10 12:09:40.876734 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.876699    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c8a24e\"/\"noten-model-c8a24e-kserve-self-signed-certs\""
Sep 10 12:09:40.882916 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.882894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:09:40.995949 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.995915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.996131 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.995965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbv7h\" (UniqueName: \"kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.996131 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.995991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.996131 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.996016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.996131 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.996052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:40.996131 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:40.996078    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.096899 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.096866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.096916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbv7h\" (UniqueName: \"kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.096953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.096979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.097024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.097063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097369 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.097343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.097503    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.097657 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.097507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.099414 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.099392    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.099660 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.099642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.106954 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.106934    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbv7h\" (UniqueName: \"kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h\") pod \"noten-model-c8a24e-kserve-579bf49bcf-z8v4c\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") " pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.183973 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.183944    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:41.459053 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:41.459013    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:09:41.461575 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:09:41.461542    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64571133_3d1f_41bc_8f15_46a4db479e43.slice/crio-00b5a4e298744f3721fd980a17b69a97a90ea7d2288a4ec995b56c12d0d7e2f3 WatchSource:0}: Error finding container 00b5a4e298744f3721fd980a17b69a97a90ea7d2288a4ec995b56c12d0d7e2f3: Status 404 returned error can't find the container with id 00b5a4e298744f3721fd980a17b69a97a90ea7d2288a4ec995b56c12d0d7e2f3
Sep 10 12:09:42.259959 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:42.259923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" event={"ID":"64571133-3d1f-41bc-8f15-46a4db479e43","Type":"ContainerStarted","Data":"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"}
Sep 10 12:09:42.259959 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:42.259962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" event={"ID":"64571133-3d1f-41bc-8f15-46a4db479e43","Type":"ContainerStarted","Data":"00b5a4e298744f3721fd980a17b69a97a90ea7d2288a4ec995b56c12d0d7e2f3"}
Sep 10 12:09:42.277817 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:42.277725    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" podStartSLOduration=2.277707258 podStartE2EDuration="2.277707258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:42.277011445 +0000 UTC m=+1083.856938831" watchObservedRunningTime="2026-09-10 12:09:42.277707258 +0000 UTC m=+1083.857634645"
Sep 10 12:09:44.414860 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.414674    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:44.415469 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.415433    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" podUID="259d0dfb-d353-4deb-9c02-7d8b0af39422" containerName="payload-processing" containerID="cri-o://5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5" gracePeriod=30
Sep 10 12:09:44.599206 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.599174    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"]
Sep 10 12:09:44.603621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.603595    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0f6353c1-l8w9l"]
Sep 10 12:09:44.670044 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.669986    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:44.835695 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.835664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume\") pod \"259d0dfb-d353-4deb-9c02-7d8b0af39422\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") "
Sep 10 12:09:44.835695 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.835697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89lcc\" (UniqueName: \"kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc\") pod \"259d0dfb-d353-4deb-9c02-7d8b0af39422\" (UID: \"259d0dfb-d353-4deb-9c02-7d8b0af39422\") "
Sep 10 12:09:44.836011 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.835991    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "259d0dfb-d353-4deb-9c02-7d8b0af39422" (UID: "259d0dfb-d353-4deb-9c02-7d8b0af39422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:44.837966 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.837942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc" (OuterVolumeSpecName: "kube-api-access-89lcc") pod "259d0dfb-d353-4deb-9c02-7d8b0af39422" (UID: "259d0dfb-d353-4deb-9c02-7d8b0af39422"). InnerVolumeSpecName "kube-api-access-89lcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:44.936398 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.936332    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259d0dfb-d353-4deb-9c02-7d8b0af39422-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:44.936398 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.936354    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lcc\" (UniqueName: \"kubernetes.io/projected/259d0dfb-d353-4deb-9c02-7d8b0af39422-kube-api-access-89lcc\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:44.946492 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:44.946462    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d592e33-9bd1-43de-b16e-74ef1c8e0084" path="/var/lib/kubelet/pods/7d592e33-9bd1-43de-b16e-74ef1c8e0084/volumes"
Sep 10 12:09:45.277235 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.277151    2573 generic.go:356] "Generic (PLEG): container finished" podID="259d0dfb-d353-4deb-9c02-7d8b0af39422" containerID="5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5" exitCode=0
Sep 10 12:09:45.277235 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.277196    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"
Sep 10 12:09:45.277235 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.277230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" event={"ID":"259d0dfb-d353-4deb-9c02-7d8b0af39422","Type":"ContainerDied","Data":"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"}
Sep 10 12:09:45.277476 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.277281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589" event={"ID":"259d0dfb-d353-4deb-9c02-7d8b0af39422","Type":"ContainerDied","Data":"83b5e3a45a04f946e7de720e620ed9fab7040fce23b97c2a25cfddca2dccbb26"}
Sep 10 12:09:45.277476 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.277338    2573 scope.go:117] "RemoveContainer" containerID="5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"
Sep 10 12:09:45.286441 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.286413    2573 scope.go:117] "RemoveContainer" containerID="5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"
Sep 10 12:09:45.286719 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:45.286701    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5\": container with ID starting with 5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5 not found: ID does not exist" containerID="5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"
Sep 10 12:09:45.286770 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.286730    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5"} err="failed to get container status \"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5\": rpc error: code = NotFound desc = could not find container \"5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5\": container with ID starting with 5f08ecfd02669c7ebad47ce16363dd3ff798989b20eaa037e245bc2c046be9b5 not found: ID does not exist"
Sep 10 12:09:45.296227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.296205    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:45.300471 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:45.300447    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0f6353c1-9c759dc5c-ts589"]
Sep 10 12:09:46.724237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:46.724201    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:09:46.724710 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:46.724444    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" podUID="654c0fb7-da99-4490-83a0-429d836506d7" containerName="payload-processing" containerID="cri-o://52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3" gracePeriod=30
Sep 10 12:09:46.946632 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:46.946600    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259d0dfb-d353-4deb-9c02-7d8b0af39422" path="/var/lib/kubelet/pods/259d0dfb-d353-4deb-9c02-7d8b0af39422/volumes"
Sep 10 12:09:46.986479 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:46.986423    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:09:47.157148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.157112    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume\") pod \"654c0fb7-da99-4490-83a0-429d836506d7\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") "
Sep 10 12:09:47.157401 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.157271    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gkm\" (UniqueName: \"kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm\") pod \"654c0fb7-da99-4490-83a0-429d836506d7\" (UID: \"654c0fb7-da99-4490-83a0-429d836506d7\") "
Sep 10 12:09:47.157514 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.157488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "654c0fb7-da99-4490-83a0-429d836506d7" (UID: "654c0fb7-da99-4490-83a0-429d836506d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:09:47.159566 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.159544    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm" (OuterVolumeSpecName: "kube-api-access-42gkm") pod "654c0fb7-da99-4490-83a0-429d836506d7" (UID: "654c0fb7-da99-4490-83a0-429d836506d7"). InnerVolumeSpecName "kube-api-access-42gkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:09:47.258388 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.258297    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gkm\" (UniqueName: \"kubernetes.io/projected/654c0fb7-da99-4490-83a0-429d836506d7-kube-api-access-42gkm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:47.258388 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.258336    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654c0fb7-da99-4490-83a0-429d836506d7-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:09:47.289423 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.289389    2573 generic.go:356] "Generic (PLEG): container finished" podID="654c0fb7-da99-4490-83a0-429d836506d7" containerID="52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3" exitCode=0
Sep 10 12:09:47.289556 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.289445    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"
Sep 10 12:09:47.289556 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.289484    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" event={"ID":"654c0fb7-da99-4490-83a0-429d836506d7","Type":"ContainerDied","Data":"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"}
Sep 10 12:09:47.289556 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.289539    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw" event={"ID":"654c0fb7-da99-4490-83a0-429d836506d7","Type":"ContainerDied","Data":"dd14af12524d80594f75a867bd6356144fae04f691e0ebb6eb619d6355fd949d"}
Sep 10 12:09:47.289702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.289564    2573 scope.go:117] "RemoveContainer" containerID="52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"
Sep 10 12:09:47.298810 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.298791    2573 scope.go:117] "RemoveContainer" containerID="52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"
Sep 10 12:09:47.299083 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:09:47.299062    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3\": container with ID starting with 52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3 not found: ID does not exist" containerID="52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"
Sep 10 12:09:47.299129 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.299093    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3"} err="failed to get container status \"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3\": rpc error: code = NotFound desc = could not find container \"52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3\": container with ID starting with 52d5ef0170a6184473c4fad85e9b1d02f1ff732ef6656e7ef766d4fae8594ce3 not found: ID does not exist"
Sep 10 12:09:47.314401 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.314375    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:09:47.319785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:47.319765    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-425dc7-699954b99c-jgwjw"]
Sep 10 12:09:48.947658 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:48.947630    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654c0fb7-da99-4490-83a0-429d836506d7" path="/var/lib/kubelet/pods/654c0fb7-da99-4490-83a0-429d836506d7/volumes"
Sep 10 12:09:51.184275 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:51.184235    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:51.184700 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:51.184291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:51.197249 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:51.197218    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:51.322614 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:51.322576    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:09:55.767147 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:09:55.767112    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:10:01.488738 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.488700    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489084    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259d0dfb-d353-4deb-9c02-7d8b0af39422" containerName="payload-processing"
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489096    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="259d0dfb-d353-4deb-9c02-7d8b0af39422" containerName="payload-processing"
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489112    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654c0fb7-da99-4490-83a0-429d836506d7" containerName="payload-processing"
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489117    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="654c0fb7-da99-4490-83a0-429d836506d7" containerName="payload-processing"
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489177    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="259d0dfb-d353-4deb-9c02-7d8b0af39422" containerName="payload-processing"
Sep 10 12:10:01.489353 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.489186    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="654c0fb7-da99-4490-83a0-429d836506d7" containerName="payload-processing"
Sep 10 12:10:01.493178 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.493161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.497392 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.497368    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjsm9\""
Sep 10 12:10:01.498159 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.498138    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-c07be0-serving-cert\""
Sep 10 12:10:01.498520 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.498503    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:10:01.518148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.518121    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:10:01.582641 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.582610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.582806 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.582651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.582806 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.582684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.582806 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.582746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67kx\" (UniqueName: \"kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.683875 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.683843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.684059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.683884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.684059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.683908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.684059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.683951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67kx\" (UniqueName: \"kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.684059 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:01.684029    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-serving-cert: secret "maas-api-e2e-shared-a-w3-c07be0-serving-cert" not found
Sep 10 12:10:01.684283 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:01.684120    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls podName:f6ac4800-1316-432c-9a21-94c17661665e nodeName:}" failed. No retries permitted until 2026-09-10 12:10:02.184098593 +0000 UTC m=+1103.764025965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" (UID: "f6ac4800-1316-432c-9a21-94c17661665e") : secret "maas-api-e2e-shared-a-w3-c07be0-serving-cert" not found
Sep 10 12:10:01.684666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.684648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.686573 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.686550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:01.693717 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:01.693693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67kx\" (UniqueName: \"kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:02.188666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.188625    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:02.191938 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.191904    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:10:02.192070 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.191953    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:02.195512 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.195495    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.199176 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.199154    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c07be0-dockercfg-x7vqs\""
Sep 10 12:10:02.199359 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.199338    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c07be0\""
Sep 10 12:10:02.208691 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.208673    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:10:02.290237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.290202    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.290427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.290262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9djq\" (UniqueName: \"kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.391838 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.391808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.392032 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.391852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9djq\" (UniqueName: \"kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.392583 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.392557    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.403573 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.403554    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:02.404621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.404594    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9djq\" (UniqueName: \"kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq\") pod \"payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.510840 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.510813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:02.677887 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.677853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:10:02.679795 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:02.679766    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ac4800_1316_432c_9a21_94c17661665e.slice/crio-7f0017198fec2df6bc1aac0663ead837cd8f471c7269e9609e5c6c7e154552b8 WatchSource:0}: Error finding container 7f0017198fec2df6bc1aac0663ead837cd8f471c7269e9609e5c6c7e154552b8: Status 404 returned error can't find the container with id 7f0017198fec2df6bc1aac0663ead837cd8f471c7269e9609e5c6c7e154552b8
Sep 10 12:10:02.784748 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:02.784716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:10:02.785982 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:02.785956    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee6ab99_583e_4faf_9abe_09959e5fbb34.slice/crio-ce05b893085aceaa63a94c27608fa9fdcd66b9346442d424572218ffbd847c75 WatchSource:0}: Error finding container ce05b893085aceaa63a94c27608fa9fdcd66b9346442d424572218ffbd847c75: Status 404 returned error can't find the container with id ce05b893085aceaa63a94c27608fa9fdcd66b9346442d424572218ffbd847c75
Sep 10 12:10:03.371998 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:03.371969    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" event={"ID":"2ee6ab99-583e-4faf-9abe-09959e5fbb34","Type":"ContainerStarted","Data":"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"}
Sep 10 12:10:03.372123 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:03.372033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" event={"ID":"2ee6ab99-583e-4faf-9abe-09959e5fbb34","Type":"ContainerStarted","Data":"ce05b893085aceaa63a94c27608fa9fdcd66b9346442d424572218ffbd847c75"}
Sep 10 12:10:03.372123 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:03.372067    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:03.373369 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:03.373347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" event={"ID":"f6ac4800-1316-432c-9a21-94c17661665e","Type":"ContainerStarted","Data":"7f0017198fec2df6bc1aac0663ead837cd8f471c7269e9609e5c6c7e154552b8"}
Sep 10 12:10:04.381416 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:04.381378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" event={"ID":"f6ac4800-1316-432c-9a21-94c17661665e","Type":"ContainerStarted","Data":"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"}
Sep 10 12:10:04.381907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:04.381453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:04.400411 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:04.400367    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" podStartSLOduration=2.741178549 podStartE2EDuration="3.400353103s" totalImagesPullingTime="659.174554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:04.399671946 +0000 UTC m=+1105.979599333" watchObservedRunningTime="2026-09-10 12:10:04.400353103 +0000 UTC m=+1105.980280488"
Sep 10 12:10:04.401177 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:04.401150    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" podStartSLOduration=2.401142296 podStartE2EDuration="2.401142296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:03.391263134 +0000 UTC m=+1104.971190522" watchObservedRunningTime="2026-09-10 12:10:04.401142296 +0000 UTC m=+1105.981069681"
Sep 10 12:10:10.393405 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:10.393374    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:10:10.791535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:10.791441    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" podUID="64571133-3d1f-41bc-8f15-46a4db479e43" containerName="main" containerID="cri-o://5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e" gracePeriod=45
Sep 10 12:10:11.057039 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.056971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:10:11.175032 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.174996    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175032 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175040    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175283 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175070    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175283 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175137    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175283 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175186    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175283 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175232    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbv7h\" (UniqueName: \"kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h\") pod \"64571133-3d1f-41bc-8f15-46a4db479e43\" (UID: \"64571133-3d1f-41bc-8f15-46a4db479e43\") "
Sep 10 12:10:11.175491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175335    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home" (OuterVolumeSpecName: "home") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:10:11.175491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175349    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache" (OuterVolumeSpecName: "model-cache") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:10:11.175586 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175572    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.175630 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175592    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.175630 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.175590    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:10:11.177533 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.177501    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:10:11.177660 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.177561    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm" (OuterVolumeSpecName: "dshm") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:10:11.177831 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.177810    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h" (OuterVolumeSpecName: "kube-api-access-lbv7h") pod "64571133-3d1f-41bc-8f15-46a4db479e43" (UID: "64571133-3d1f-41bc-8f15-46a4db479e43"). InnerVolumeSpecName "kube-api-access-lbv7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:10:11.276800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.276755    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64571133-3d1f-41bc-8f15-46a4db479e43-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.276800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.276797    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.276800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.276808    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64571133-3d1f-41bc-8f15-46a4db479e43-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.277031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.276820    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbv7h\" (UniqueName: \"kubernetes.io/projected/64571133-3d1f-41bc-8f15-46a4db479e43-kube-api-access-lbv7h\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:10:11.419970 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.419938    2573 generic.go:356] "Generic (PLEG): container finished" podID="64571133-3d1f-41bc-8f15-46a4db479e43" containerID="5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e" exitCode=0
Sep 10 12:10:11.420415 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.419981    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"
Sep 10 12:10:11.420415 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.420019    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" event={"ID":"64571133-3d1f-41bc-8f15-46a4db479e43","Type":"ContainerDied","Data":"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"}
Sep 10 12:10:11.420415 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.420062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c" event={"ID":"64571133-3d1f-41bc-8f15-46a4db479e43","Type":"ContainerDied","Data":"00b5a4e298744f3721fd980a17b69a97a90ea7d2288a4ec995b56c12d0d7e2f3"}
Sep 10 12:10:11.420415 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.420081    2573 scope.go:117] "RemoveContainer" containerID="5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"
Sep 10 12:10:11.429677 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.429661    2573 scope.go:117] "RemoveContainer" containerID="5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"
Sep 10 12:10:11.429938 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:11.429917    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e\": container with ID starting with 5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e not found: ID does not exist" containerID="5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"
Sep 10 12:10:11.429999 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.429951    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e"} err="failed to get container status \"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e\": rpc error: code = NotFound desc = could not find container \"5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e\": container with ID starting with 5af00f9078795fd2ad10de45e69085a38611e67312027d8225263d67c289656e not found: ID does not exist"
Sep 10 12:10:11.446649 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.446626    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:10:11.449755 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:11.449733    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c8a24e/noten-model-c8a24e-kserve-579bf49bcf-z8v4c"]
Sep 10 12:10:12.947716 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:12.947680    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64571133-3d1f-41bc-8f15-46a4db479e43" path="/var/lib/kubelet/pods/64571133-3d1f-41bc-8f15-46a4db479e43/volumes"
Sep 10 12:10:14.213433 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.213397    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:10:14.213812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.213800    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64571133-3d1f-41bc-8f15-46a4db479e43" containerName="main"
Sep 10 12:10:14.213812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.213811    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="64571133-3d1f-41bc-8f15-46a4db479e43" containerName="main"
Sep 10 12:10:14.213901 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.213879    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="64571133-3d1f-41bc-8f15-46a4db479e43" containerName="main"
Sep 10 12:10:14.218958 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.218937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.221892 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.221860    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-639beb-dockercfg-s9kh8\""
Sep 10 12:10:14.222015 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.221898    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-639beb\""
Sep 10 12:10:14.231034 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.231013    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:10:14.279080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.279058    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:10:14.282733 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.282717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.294978 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.294958    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:10:14.301458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.301432    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdw4g\" (UniqueName: \"kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.301532 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.301478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.383822 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.383802    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:10:14.402048 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.402025    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.402284 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.402264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw4g\" (UniqueName: \"kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.402411 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.402391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.402480 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.402431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzdh\" (UniqueName: \"kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.403129 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.403112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.416566 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.416543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw4g\" (UniqueName: \"kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g\") pod \"payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.503480 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.503398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzdh\" (UniqueName: \"kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.503480 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.503472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.504075 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.504055    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.511785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.511761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzdh\" (UniqueName: \"kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh\") pod \"payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.529642 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.529616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:14.592594 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.592557    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:14.808403 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.808379    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:10:14.810030 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:14.810009    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc20928_b32c_4e29_998f_737fc3f08ce1.slice/crio-8e7a8afbafe4910c66818feec76610a8d8975bb2342f7e18a397d0e67284e217 WatchSource:0}: Error finding container 8e7a8afbafe4910c66818feec76610a8d8975bb2342f7e18a397d0e67284e217: Status 404 returned error can't find the container with id 8e7a8afbafe4910c66818feec76610a8d8975bb2342f7e18a397d0e67284e217
Sep 10 12:10:14.869165 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:14.869146    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:10:14.877356 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:14.877332    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c0102_933c_4a71_8b92_eb8395702dc4.slice/crio-5f1b2fd80f125be4f9cc5a21fc0f1f340108c53fa4243178fc65326ed06813f0 WatchSource:0}: Error finding container 5f1b2fd80f125be4f9cc5a21fc0f1f340108c53fa4243178fc65326ed06813f0: Status 404 returned error can't find the container with id 5f1b2fd80f125be4f9cc5a21fc0f1f340108c53fa4243178fc65326ed06813f0
Sep 10 12:10:15.444648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.444609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" event={"ID":"5cc20928-b32c-4e29-998f-737fc3f08ce1","Type":"ContainerStarted","Data":"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"}
Sep 10 12:10:15.445028 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.444686    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:15.445028 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.444711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" event={"ID":"5cc20928-b32c-4e29-998f-737fc3f08ce1","Type":"ContainerStarted","Data":"8e7a8afbafe4910c66818feec76610a8d8975bb2342f7e18a397d0e67284e217"}
Sep 10 12:10:15.446868 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.446838    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" event={"ID":"f26c0102-933c-4a71-8b92-eb8395702dc4","Type":"ContainerStarted","Data":"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"}
Sep 10 12:10:15.446985 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.446883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" event={"ID":"f26c0102-933c-4a71-8b92-eb8395702dc4","Type":"ContainerStarted","Data":"5f1b2fd80f125be4f9cc5a21fc0f1f340108c53fa4243178fc65326ed06813f0"}
Sep 10 12:10:15.446985 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.446904    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:15.463794 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.463555    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" podStartSLOduration=1.463536009 podStartE2EDuration="1.463536009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:15.463215751 +0000 UTC m=+1117.043143136" watchObservedRunningTime="2026-09-10 12:10:15.463536009 +0000 UTC m=+1117.043463400"
Sep 10 12:10:15.482616 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:15.482567    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" podStartSLOduration=1.482548387 podStartE2EDuration="1.482548387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:15.48165666 +0000 UTC m=+1117.061584056" watchObservedRunningTime="2026-09-10 12:10:15.482548387 +0000 UTC m=+1117.062475777"
Sep 10 12:10:26.454483 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:26.454453    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:10:26.454912 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:26.454525    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:10:30.690056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.690015    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:10:30.693690 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.693663    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.697951 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.697925    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7ed5ac-serving-cert\""
Sep 10 12:10:30.709952 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.709924    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:10:30.858390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.858351    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.858390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.858391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x24m\" (UniqueName: \"kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.858656 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.858479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.858656 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.858559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.959427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.959271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.959427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.959348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.959427 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.959420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.959718 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:30.959424    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-serving-cert: secret "maas-api-e2e-shared-b-w3-7ed5ac-serving-cert" not found
Sep 10 12:10:30.959718 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.959437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x24m\" (UniqueName: \"kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.959718 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:30.959494    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls podName:65154c05-b8ac-4be1-9f60-da804745deb3 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:31.459474228 +0000 UTC m=+1133.039401592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls") pod "maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" (UID: "65154c05-b8ac-4be1-9f60-da804745deb3") : secret "maas-api-e2e-shared-b-w3-7ed5ac-serving-cert" not found
Sep 10 12:10:30.960094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.960070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.962014 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.961989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:30.971653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:30.971625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x24m\" (UniqueName: \"kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:31.464110 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:31.464062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:31.466672 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:31.466649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:31.605927 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:31.605893    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:31.881389 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:31.881362    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:10:31.883180 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:31.883151    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65154c05_b8ac_4be1_9f60_da804745deb3.slice/crio-347e1f5cd565808aa0ab65cacdb2e39cd9905262e395deb309b5cdce93989084 WatchSource:0}: Error finding container 347e1f5cd565808aa0ab65cacdb2e39cd9905262e395deb309b5cdce93989084: Status 404 returned error can't find the container with id 347e1f5cd565808aa0ab65cacdb2e39cd9905262e395deb309b5cdce93989084
Sep 10 12:10:32.547885 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:32.547844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" event={"ID":"65154c05-b8ac-4be1-9f60-da804745deb3","Type":"ContainerStarted","Data":"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"}
Sep 10 12:10:32.548089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:32.547899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" event={"ID":"65154c05-b8ac-4be1-9f60-da804745deb3","Type":"ContainerStarted","Data":"347e1f5cd565808aa0ab65cacdb2e39cd9905262e395deb309b5cdce93989084"}
Sep 10 12:10:32.548089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:32.547937    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:32.569752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:32.569651    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" podStartSLOduration=2.140836659 podStartE2EDuration="2.569634971s" totalImagesPullingTime="428.798312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:32.569171597 +0000 UTC m=+1134.149098985" watchObservedRunningTime="2026-09-10 12:10:32.569634971 +0000 UTC m=+1134.149562367"
Sep 10 12:10:38.559246 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:38.559213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:10:42.687112 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.687081    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:10:42.692506 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.692483    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.696870 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.696849    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c66562-serving-cert\""
Sep 10 12:10:42.721172 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.721144    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:10:42.769089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.769059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.769261 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.769100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlzp\" (UniqueName: \"kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.769261 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.769192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.769410 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.769293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.869926 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.869891    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.870097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.869948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlzp\" (UniqueName: \"kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.870097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.870001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.870097 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:42.870045    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-serving-cert: secret "maas-api-e2e-shared-b-w2-c66562-serving-cert" not found
Sep 10 12:10:42.870097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.870069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.870358 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:42.870124    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls podName:aa5fe72a-9ca3-4d5f-90d1-e5559b709372 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:43.370101904 +0000 UTC m=+1144.950029271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372") : secret "maas-api-e2e-shared-b-w2-c66562-serving-cert" not found
Sep 10 12:10:42.871223 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.871199    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.872994 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.872975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:42.891547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:42.891517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlzp\" (UniqueName: \"kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:43.374396 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.374362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:43.374599 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:43.374543    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-serving-cert: secret "maas-api-e2e-shared-b-w2-c66562-serving-cert" not found
Sep 10 12:10:43.374663 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:10:43.374623    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls podName:aa5fe72a-9ca3-4d5f-90d1-e5559b709372 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:44.374602467 +0000 UTC m=+1145.954529834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372") : secret "maas-api-e2e-shared-b-w2-c66562-serving-cert" not found
Sep 10 12:10:43.708167 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.708130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:10:43.711943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.711927    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.715139 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.715114    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c66562-dockercfg-w9lw4\""
Sep 10 12:10:43.715249 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.715198    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c66562\""
Sep 10 12:10:43.723333 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.723295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:10:43.778160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.778124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.778160 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.778158    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpbm\" (UniqueName: \"kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.879030 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.878996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.879030 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.879029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpbm\" (UniqueName: \"kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.879733 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.879710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:43.891981 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:43.891950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpbm\" (UniqueName: \"kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm\") pod \"payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:44.023871 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.023781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:44.300431 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.300405    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:10:44.302949 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:44.302922    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f6ee90_5df9_4116_8439_83e2abd5678a.slice/crio-3ae553d03edd9794e571b7c5eb5d789ae0ab2a4748ff251009b1192198627694 WatchSource:0}: Error finding container 3ae553d03edd9794e571b7c5eb5d789ae0ab2a4748ff251009b1192198627694: Status 404 returned error can't find the container with id 3ae553d03edd9794e571b7c5eb5d789ae0ab2a4748ff251009b1192198627694
Sep 10 12:10:44.383807 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.383776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:44.386079 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.386053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:44.503476 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.503437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:44.615901 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.615871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" event={"ID":"42f6ee90-5df9-4116-8439-83e2abd5678a","Type":"ContainerStarted","Data":"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"}
Sep 10 12:10:44.616055 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.615919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" event={"ID":"42f6ee90-5df9-4116-8439-83e2abd5678a","Type":"ContainerStarted","Data":"3ae553d03edd9794e571b7c5eb5d789ae0ab2a4748ff251009b1192198627694"}
Sep 10 12:10:44.616055 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.615951    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:10:44.651594 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.651543    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" podStartSLOduration=1.651526593 podStartE2EDuration="1.651526593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:44.648265621 +0000 UTC m=+1146.228193006" watchObservedRunningTime="2026-09-10 12:10:44.651526593 +0000 UTC m=+1146.231453978"
Sep 10 12:10:44.828838 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:44.828758    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:10:44.831159 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:10:44.831134    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5fe72a_9ca3_4d5f_90d1_e5559b709372.slice/crio-751006489c73b5285e410dcfb58f11dbd7dae042bcf7980c9f4fe099c0176831 WatchSource:0}: Error finding container 751006489c73b5285e410dcfb58f11dbd7dae042bcf7980c9f4fe099c0176831: Status 404 returned error can't find the container with id 751006489c73b5285e410dcfb58f11dbd7dae042bcf7980c9f4fe099c0176831
Sep 10 12:10:45.622866 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:45.622826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" event={"ID":"aa5fe72a-9ca3-4d5f-90d1-e5559b709372","Type":"ContainerStarted","Data":"751006489c73b5285e410dcfb58f11dbd7dae042bcf7980c9f4fe099c0176831"}
Sep 10 12:10:46.631019 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:46.630980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" event={"ID":"aa5fe72a-9ca3-4d5f-90d1-e5559b709372","Type":"ContainerStarted","Data":"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"}
Sep 10 12:10:46.631416 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:46.631117    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:46.674985 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:46.674933    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" podStartSLOduration=3.585046936 podStartE2EDuration="4.674911392s" totalImagesPullingTime="1.089864456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:46.673504308 +0000 UTC m=+1148.253431694" watchObservedRunningTime="2026-09-10 12:10:46.674911392 +0000 UTC m=+1148.254838777"
Sep 10 12:10:52.643107 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:52.643074    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:10:55.624873 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:10:55.624844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:11:26.667449 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.667415    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:11:26.672490 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.672470    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.675340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.675317    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e74b3f66-dockercfg-7c5t8\""
Sep 10 12:11:26.675456 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.675358    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e74b3f66\""
Sep 10 12:11:26.677089 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.677062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.677214 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.677184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4kt\" (UniqueName: \"kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.694983 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.694956    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:11:26.778471 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.778442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4kt\" (UniqueName: \"kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.778646 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.778552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.779243 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.779222    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.790157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.790136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4kt\" (UniqueName: \"kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt\") pod \"payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") " pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:26.984141 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:26.984060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:27.258895 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:27.258870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:11:27.260993 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:11:27.260962    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178418d0_8adc_40e4_bf43_08cc9a3b1749.slice/crio-7f4f2b1f596fc3910418c61e36b34999d2ab2985fffe07b799e322eb831fc2c5 WatchSource:0}: Error finding container 7f4f2b1f596fc3910418c61e36b34999d2ab2985fffe07b799e322eb831fc2c5: Status 404 returned error can't find the container with id 7f4f2b1f596fc3910418c61e36b34999d2ab2985fffe07b799e322eb831fc2c5
Sep 10 12:11:27.847552 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:27.847517    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" event={"ID":"178418d0-8adc-40e4-bf43-08cc9a3b1749","Type":"ContainerStarted","Data":"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"}
Sep 10 12:11:27.848034 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:27.847574    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:27.848034 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:27.847591    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" event={"ID":"178418d0-8adc-40e4-bf43-08cc9a3b1749","Type":"ContainerStarted","Data":"7f4f2b1f596fc3910418c61e36b34999d2ab2985fffe07b799e322eb831fc2c5"}
Sep 10 12:11:27.866446 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:27.866393    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" podStartSLOduration=1.866377374 podStartE2EDuration="1.866377374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:27.863952253 +0000 UTC m=+1189.443879638" watchObservedRunningTime="2026-09-10 12:11:27.866377374 +0000 UTC m=+1189.446304814"
Sep 10 12:11:38.855328 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:38.855283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:11:38.972964 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:38.972932    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:11:38.974792 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:38.974767    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:11:39.069597 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:39.069561    2573 scope.go:117] "RemoveContainer" containerID="9518d1a3870400ba2b4b1a85b754aef4bdbdb336739408c5c98a8bcb82edada6"
Sep 10 12:11:39.078813 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:39.078793    2573 scope.go:117] "RemoveContainer" containerID="8c7f774dfc5930675e8d487315cfff7b7c3163ba340e44bc4791baade29f0001"
Sep 10 12:11:47.317171 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.317137    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:11:47.322080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.322063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.326666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.326636    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"openshift-service-ca.crt\""
Sep 10 12:11:47.326666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.326656    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"default-dockercfg-2gvdn\""
Sep 10 12:11:47.326666 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.326663    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"auth-test-model-c07be0-kserve-self-signed-certs\""
Sep 10 12:11:47.326863 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.326685    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"kube-root-ca.crt\""
Sep 10 12:11:47.332368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.332344    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:11:47.372406 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.372547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.372547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.372547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwb4p\" (UniqueName: \"kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.372729 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.372729 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.372695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473282 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473282 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473432    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwb4p\" (UniqueName: \"kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473716 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473772 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473730    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473827 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.473879 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.473832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.476074 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.476053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.476312 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.476277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.483069 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.483045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwb4p\" (UniqueName: \"kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p\") pod \"auth-test-model-c07be0-kserve-789f4767c9-wwlcx\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.633502 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.633420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:47.910210 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.910187    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:11:47.911840 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:11:47.911814    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20ee623c_115e_424a_b897_2f4a70ee5548.slice/crio-4f01f0c9a045a2e657b1a3357fc3ace79fe258e34a7b695bb1e9ccd20863646e WatchSource:0}: Error finding container 4f01f0c9a045a2e657b1a3357fc3ace79fe258e34a7b695bb1e9ccd20863646e: Status 404 returned error can't find the container with id 4f01f0c9a045a2e657b1a3357fc3ace79fe258e34a7b695bb1e9ccd20863646e
Sep 10 12:11:47.956045 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:47.956018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" event={"ID":"20ee623c-115e-424a-b897-2f4a70ee5548","Type":"ContainerStarted","Data":"4f01f0c9a045a2e657b1a3357fc3ace79fe258e34a7b695bb1e9ccd20863646e"}
Sep 10 12:11:48.962154 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:48.962116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" event={"ID":"20ee623c-115e-424a-b897-2f4a70ee5548","Type":"ContainerStarted","Data":"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"}
Sep 10 12:11:48.980547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:48.980502    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" podStartSLOduration=1.980486362 podStartE2EDuration="1.980486362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:48.978838465 +0000 UTC m=+1210.558765852" watchObservedRunningTime="2026-09-10 12:11:48.980486362 +0000 UTC m=+1210.560413783"
Sep 10 12:11:57.634425 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:57.634383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:57.634835 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:57.634464    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:57.648709 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:57.648688    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:11:58.016417 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:11:58.016389    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:13:16.309598 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.308942    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:13:16.309598 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.309197    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" podUID="178418d0-8adc-40e4-bf43-08cc9a3b1749" containerName="payload-processing" containerID="cri-o://e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934" gracePeriod=30
Sep 10 12:13:16.564614 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.564541    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:13:16.738036 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.737998    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume\") pod \"178418d0-8adc-40e4-bf43-08cc9a3b1749\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") "
Sep 10 12:13:16.738218 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.738115    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg4kt\" (UniqueName: \"kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt\") pod \"178418d0-8adc-40e4-bf43-08cc9a3b1749\" (UID: \"178418d0-8adc-40e4-bf43-08cc9a3b1749\") "
Sep 10 12:13:16.738426 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.738402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "178418d0-8adc-40e4-bf43-08cc9a3b1749" (UID: "178418d0-8adc-40e4-bf43-08cc9a3b1749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:13:16.740323 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.740282    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt" (OuterVolumeSpecName: "kube-api-access-cg4kt") pod "178418d0-8adc-40e4-bf43-08cc9a3b1749" (UID: "178418d0-8adc-40e4-bf43-08cc9a3b1749"). InnerVolumeSpecName "kube-api-access-cg4kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:13:16.839843 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.839767    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/178418d0-8adc-40e4-bf43-08cc9a3b1749-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:13:16.839843 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:16.839791    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg4kt\" (UniqueName: \"kubernetes.io/projected/178418d0-8adc-40e4-bf43-08cc9a3b1749-kube-api-access-cg4kt\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:13:17.413656 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.413622    2573 generic.go:356] "Generic (PLEG): container finished" podID="178418d0-8adc-40e4-bf43-08cc9a3b1749" containerID="e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934" exitCode=0
Sep 10 12:13:17.414047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.413672    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"
Sep 10 12:13:17.414047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.413696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" event={"ID":"178418d0-8adc-40e4-bf43-08cc9a3b1749","Type":"ContainerDied","Data":"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"}
Sep 10 12:13:17.414047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.413738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm" event={"ID":"178418d0-8adc-40e4-bf43-08cc9a3b1749","Type":"ContainerDied","Data":"7f4f2b1f596fc3910418c61e36b34999d2ab2985fffe07b799e322eb831fc2c5"}
Sep 10 12:13:17.414047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.413759    2573 scope.go:117] "RemoveContainer" containerID="e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"
Sep 10 12:13:17.423102 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.423078    2573 scope.go:117] "RemoveContainer" containerID="e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"
Sep 10 12:13:17.423382 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:13:17.423364    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934\": container with ID starting with e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934 not found: ID does not exist" containerID="e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"
Sep 10 12:13:17.423445 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.423390    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934"} err="failed to get container status \"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934\": rpc error: code = NotFound desc = could not find container \"e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934\": container with ID starting with e1339280911a676cad4d6a7e5980f03c6559b229763cdff422637a2685a46934 not found: ID does not exist"
Sep 10 12:13:17.431380 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.431349    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:13:17.435460 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:17.435439    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e74b3f66-fd999cd57-bnsxm"]
Sep 10 12:13:18.947613 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:13:18.947584    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178418d0-8adc-40e4-bf43-08cc9a3b1749" path="/var/lib/kubelet/pods/178418d0-8adc-40e4-bf43-08cc9a3b1749/volumes"
Sep 10 12:14:45.861510 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:14:45.861476    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:15:00.882502 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:00.882460    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" podUID="20ee623c-115e-424a-b897-2f4a70ee5548" containerName="main" containerID="cri-o://576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581" gracePeriod=45
Sep 10 12:15:01.139051 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.139025    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:15:01.229082 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229045    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwb4p\" (UniqueName: \"kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229197    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229272 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229225    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229515 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229290    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs\") pod \"20ee623c-115e-424a-b897-2f4a70ee5548\" (UID: \"20ee623c-115e-424a-b897-2f4a70ee5548\") "
Sep 10 12:15:01.229515 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229469    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:15:01.229515 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229484    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home" (OuterVolumeSpecName: "home") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:15:01.229515 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache" (OuterVolumeSpecName: "model-cache") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:15:01.229750 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229730    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.229812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229759    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.229812 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.229774    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.231465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.231446    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm" (OuterVolumeSpecName: "dshm") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:15:01.232092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.232070    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p" (OuterVolumeSpecName: "kube-api-access-xwb4p") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "kube-api-access-xwb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:15:01.232092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.232076    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20ee623c-115e-424a-b897-2f4a70ee5548" (UID: "20ee623c-115e-424a-b897-2f4a70ee5548"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:15:01.331282 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.331252    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwb4p\" (UniqueName: \"kubernetes.io/projected/20ee623c-115e-424a-b897-2f4a70ee5548-kube-api-access-xwb4p\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.331282 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.331280    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ee623c-115e-424a-b897-2f4a70ee5548-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.331586 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.331294    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ee623c-115e-424a-b897-2f4a70ee5548-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:15:01.950966 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.950934    2573 generic.go:356] "Generic (PLEG): container finished" podID="20ee623c-115e-424a-b897-2f4a70ee5548" containerID="576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581" exitCode=0
Sep 10 12:15:01.951455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.950971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"
Sep 10 12:15:01.951455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.950966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" event={"ID":"20ee623c-115e-424a-b897-2f4a70ee5548","Type":"ContainerDied","Data":"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"}
Sep 10 12:15:01.951455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.951087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx" event={"ID":"20ee623c-115e-424a-b897-2f4a70ee5548","Type":"ContainerDied","Data":"4f01f0c9a045a2e657b1a3357fc3ace79fe258e34a7b695bb1e9ccd20863646e"}
Sep 10 12:15:01.951455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.951116    2573 scope.go:117] "RemoveContainer" containerID="576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"
Sep 10 12:15:01.962563 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.962544    2573 scope.go:117] "RemoveContainer" containerID="576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"
Sep 10 12:15:01.962888 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:15:01.962869    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581\": container with ID starting with 576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581 not found: ID does not exist" containerID="576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"
Sep 10 12:15:01.962971 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.962900    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581"} err="failed to get container status \"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581\": rpc error: code = NotFound desc = could not find container \"576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581\": container with ID starting with 576612a9989767b961f6bf7fd22657120c678168007ed9e69ccd5bf731d41581 not found: ID does not exist"
Sep 10 12:15:01.974863 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.974842    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:15:01.979478 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:01.979458    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/auth-test-model-c07be0-kserve-789f4767c9-wwlcx"]
Sep 10 12:15:02.946787 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:02.946756    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ee623c-115e-424a-b897-2f4a70ee5548" path="/var/lib/kubelet/pods/20ee623c-115e-424a-b897-2f4a70ee5548/volumes"
Sep 10 12:15:15.400874 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.400838    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:15:15.401699 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401676    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178418d0-8adc-40e4-bf43-08cc9a3b1749" containerName="payload-processing"
Sep 10 12:15:15.401699 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401700    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="178418d0-8adc-40e4-bf43-08cc9a3b1749" containerName="payload-processing"
Sep 10 12:15:15.401888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401717    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ee623c-115e-424a-b897-2f4a70ee5548" containerName="main"
Sep 10 12:15:15.401888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401723    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ee623c-115e-424a-b897-2f4a70ee5548" containerName="main"
Sep 10 12:15:15.401888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401810    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="178418d0-8adc-40e4-bf43-08cc9a3b1749" containerName="payload-processing"
Sep 10 12:15:15.401888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.401825    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ee623c-115e-424a-b897-2f4a70ee5548" containerName="main"
Sep 10 12:15:15.405078 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.405060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.407909 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.407883    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-187e7e-serving-cert\""
Sep 10 12:15:15.422075 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.422016    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:15:15.565937 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.565903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdz5\" (UniqueName: \"kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.566112 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.565970    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.566112 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.565991    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.566112 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.566064    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.666721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.666633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.666721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.666667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.666721 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.666695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.667006 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.666758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdz5\" (UniqueName: \"kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.667006 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:15:15.666802    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-serving-cert: secret "maas-api-e2e-api-a-187e7e-serving-cert" not found
Sep 10 12:15:15.667006 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:15:15.666883    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls podName:af77607b-60b5-4ba8-a3e2-cd9a93cfb130 nodeName:}" failed. No retries permitted until 2026-09-10 12:15:16.166860923 +0000 UTC m=+1417.746788306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls") pod "maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" (UID: "af77607b-60b5-4ba8-a3e2-cd9a93cfb130") : secret "maas-api-e2e-api-a-187e7e-serving-cert" not found
Sep 10 12:15:15.667428 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.667405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.669314 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.669285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:15.679730 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:15.679710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdz5\" (UniqueName: \"kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:16.172400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:16.172365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:16.174888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:16.174861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") pod \"maas-api-e2e-api-a-187e7e-54fcc78446-67l9k\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:16.318739 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:16.318704    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:16.597104 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:16.597079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:15:16.598606 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:15:16.598578    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf77607b_60b5_4ba8_a3e2_cd9a93cfb130.slice/crio-1ad52c2c82a900b1aeb50335076920049d27fbec84ab9b0cddfe864b6492a1f3 WatchSource:0}: Error finding container 1ad52c2c82a900b1aeb50335076920049d27fbec84ab9b0cddfe864b6492a1f3: Status 404 returned error can't find the container with id 1ad52c2c82a900b1aeb50335076920049d27fbec84ab9b0cddfe864b6492a1f3
Sep 10 12:15:16.599925 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:16.599905    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:15:17.029440 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:17.029405    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" event={"ID":"af77607b-60b5-4ba8-a3e2-cd9a93cfb130","Type":"ContainerStarted","Data":"1ad52c2c82a900b1aeb50335076920049d27fbec84ab9b0cddfe864b6492a1f3"}
Sep 10 12:15:18.035621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:18.035524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" event={"ID":"af77607b-60b5-4ba8-a3e2-cd9a93cfb130","Type":"ContainerStarted","Data":"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"}
Sep 10 12:15:18.035621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:18.035593    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:24.046149 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:24.046117    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:15:24.109023 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:24.108961    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" podStartSLOduration=7.96609704 podStartE2EDuration="9.108945213s" totalImagesPullingTime="1.142848173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:18.080068142 +0000 UTC m=+1419.659995528" watchObservedRunningTime="2026-09-10 12:15:24.108945213 +0000 UTC m=+1425.688872599"
Sep 10 12:15:37.690917 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.690834    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:15:37.694605 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.694583    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.703001 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.702977    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"openshift-service-ca.crt\""
Sep 10 12:15:37.703117 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.703026    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"kube-root-ca.crt\""
Sep 10 12:15:37.703117 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.703063    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"sub-test-model-7ed5ac-kserve-self-signed-certs\""
Sep 10 12:15:37.703227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.703151    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"default-dockercfg-69mpl\""
Sep 10 12:15:37.710331 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.710288    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:15:37.782218 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.782218 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.782465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.782465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89g8\" (UniqueName: \"kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.782465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.782567 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.782504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.883889 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.883850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.883889 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.883893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89g8\" (UniqueName: \"kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.883918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.883960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.884036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.884060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884443 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.884418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884443 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.884440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.884570 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.884468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.886156 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.886126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.886530 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.886512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:37.892810 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:37.892792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89g8\" (UniqueName: \"kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8\") pod \"sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:38.004013 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:38.003923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:38.285033 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:38.284930    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:15:38.287576 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:15:38.287548    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb504447_ed4c_4074_b685_89e49c821f77.slice/crio-d947cfc0e342c7b999893bff4fc3324f6d26843361e3918e6e5fc7e30bc26378 WatchSource:0}: Error finding container d947cfc0e342c7b999893bff4fc3324f6d26843361e3918e6e5fc7e30bc26378: Status 404 returned error can't find the container with id d947cfc0e342c7b999893bff4fc3324f6d26843361e3918e6e5fc7e30bc26378
Sep 10 12:15:39.128652 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:39.128623    2573 scope.go:117] "RemoveContainer" containerID="f98f350e14e9967075a0c8d03885b5ed93df6c83cf2d504d002d22abe521904a"
Sep 10 12:15:39.155817 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:39.155788    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" event={"ID":"eb504447-ed4c-4074-b685-89e49c821f77","Type":"ContainerStarted","Data":"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"}
Sep 10 12:15:39.155933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:39.155833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" event={"ID":"eb504447-ed4c-4074-b685-89e49c821f77","Type":"ContainerStarted","Data":"d947cfc0e342c7b999893bff4fc3324f6d26843361e3918e6e5fc7e30bc26378"}
Sep 10 12:15:39.188002 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:39.187950    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" podStartSLOduration=2.187933594 podStartE2EDuration="2.187933594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:39.187643097 +0000 UTC m=+1440.767570483" watchObservedRunningTime="2026-09-10 12:15:39.187933594 +0000 UTC m=+1440.767860980"
Sep 10 12:15:45.233424 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.233387    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:15:45.238563 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.238539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.241395 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.241376    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c531cb\""
Sep 10 12:15:45.241503 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.241451    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c531cb-dockercfg-7dg6c\""
Sep 10 12:15:45.248769 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.248734    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:15:45.357250 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.357215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.357417 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.357272    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbq8r\" (UniqueName: \"kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.458108 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.458063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbq8r\" (UniqueName: \"kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.458354 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.458197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.458931 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.458905    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.466251 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.466228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbq8r\" (UniqueName: \"kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r\") pod \"payload-processing-e2e-api-b-c531cb-bb594c859-792p6\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") " pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.549845 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.549764    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:45.823508 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:45.823481    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:15:45.825636 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:15:45.825589    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f430ff2_9593_470c_85d9_048ca1f07801.slice/crio-f04af2e3961e927fadb176ca8360c46bd0eb86090edc5f79c68b2cef987ac5b8 WatchSource:0}: Error finding container f04af2e3961e927fadb176ca8360c46bd0eb86090edc5f79c68b2cef987ac5b8: Status 404 returned error can't find the container with id f04af2e3961e927fadb176ca8360c46bd0eb86090edc5f79c68b2cef987ac5b8
Sep 10 12:15:46.192900 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:46.192860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" event={"ID":"4f430ff2-9593-470c-85d9-048ca1f07801","Type":"ContainerStarted","Data":"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"}
Sep 10 12:15:46.193116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:46.192917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" event={"ID":"4f430ff2-9593-470c-85d9-048ca1f07801","Type":"ContainerStarted","Data":"f04af2e3961e927fadb176ca8360c46bd0eb86090edc5f79c68b2cef987ac5b8"}
Sep 10 12:15:46.193116 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:46.192990    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:15:46.233394 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:46.233326    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" podStartSLOduration=1.23329155 podStartE2EDuration="1.23329155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:46.232365412 +0000 UTC m=+1447.812292798" watchObservedRunningTime="2026-09-10 12:15:46.23329155 +0000 UTC m=+1447.813218938"
Sep 10 12:15:48.004750 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:48.004706    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:48.005149 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:48.004788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:48.017080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:48.017058    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:48.213737 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:48.213708    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:15:57.200925 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:15:57.200894    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:16:26.923482 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:26.923443    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:16:26.923988 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:26.923708    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" podUID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" containerName="maas-api" containerID="cri-o://4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3" gracePeriod=30
Sep 10 12:16:27.169693 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.169669    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:16:27.268808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.268721    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs\") pod \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") "
Sep 10 12:16:27.268808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.268788    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") pod \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") "
Sep 10 12:16:27.269046 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.268833    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pdz5\" (UniqueName: \"kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5\") pod \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") "
Sep 10 12:16:27.269046 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.268884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca\") pod \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\" (UID: \"af77607b-60b5-4ba8-a3e2-cd9a93cfb130\") "
Sep 10 12:16:27.269374 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.269346    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af77607b-60b5-4ba8-a3e2-cd9a93cfb130" (UID: "af77607b-60b5-4ba8-a3e2-cd9a93cfb130"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:16:27.270970 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.270940    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "af77607b-60b5-4ba8-a3e2-cd9a93cfb130" (UID: "af77607b-60b5-4ba8-a3e2-cd9a93cfb130"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:16:27.271080 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.270969    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af77607b-60b5-4ba8-a3e2-cd9a93cfb130" (UID: "af77607b-60b5-4ba8-a3e2-cd9a93cfb130"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:16:27.271411 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.271077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5" (OuterVolumeSpecName: "kube-api-access-5pdz5") pod "af77607b-60b5-4ba8-a3e2-cd9a93cfb130" (UID: "af77607b-60b5-4ba8-a3e2-cd9a93cfb130"). InnerVolumeSpecName "kube-api-access-5pdz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:16:27.370031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.369992    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:27.370031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.370025    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdz5\" (UniqueName: \"kubernetes.io/projected/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-kube-api-access-5pdz5\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:27.370031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.370037    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:27.370248 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.370049    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af77607b-60b5-4ba8-a3e2-cd9a93cfb130-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:27.414933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.414903    2573 generic.go:356] "Generic (PLEG): container finished" podID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" containerID="4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3" exitCode=0
Sep 10 12:16:27.415092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.414971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"
Sep 10 12:16:27.415092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.414996    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" event={"ID":"af77607b-60b5-4ba8-a3e2-cd9a93cfb130","Type":"ContainerDied","Data":"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"}
Sep 10 12:16:27.415092 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.415066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k" event={"ID":"af77607b-60b5-4ba8-a3e2-cd9a93cfb130","Type":"ContainerDied","Data":"1ad52c2c82a900b1aeb50335076920049d27fbec84ab9b0cddfe864b6492a1f3"}
Sep 10 12:16:27.415243 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.415104    2573 scope.go:117] "RemoveContainer" containerID="4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"
Sep 10 12:16:27.424696 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.424677    2573 scope.go:117] "RemoveContainer" containerID="4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"
Sep 10 12:16:27.424994 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:16:27.424977    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3\": container with ID starting with 4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3 not found: ID does not exist" containerID="4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"
Sep 10 12:16:27.425034 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.425002    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3"} err="failed to get container status \"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3\": rpc error: code = NotFound desc = could not find container \"4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3\": container with ID starting with 4cf3da19e0ff762439ec4646c8b27dc3dd94fe084203dd0ce1eb88f579d166b3 not found: ID does not exist"
Sep 10 12:16:27.438199 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.438178    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:16:27.440157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:27.440140    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-187e7e-54fcc78446-67l9k"]
Sep 10 12:16:28.947550 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:28.947516    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" path="/var/lib/kubelet/pods/af77607b-60b5-4ba8-a3e2-cd9a93cfb130/volumes"
Sep 10 12:16:31.080110 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:31.080079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:16:36.653281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.653247    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:16:36.653740 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.653730    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" containerName="maas-api"
Sep 10 12:16:36.653781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.653743    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" containerName="maas-api"
Sep 10 12:16:36.653818 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.653803    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="af77607b-60b5-4ba8-a3e2-cd9a93cfb130" containerName="maas-api"
Sep 10 12:16:36.658073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.658054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.661287 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.661259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"rate-test-model-c07be0-kserve-self-signed-certs\""
Sep 10 12:16:36.662431 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.662413    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"kube-root-ca.crt\""
Sep 10 12:16:36.662539 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.662517    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"default-dockercfg-2gvdn\""
Sep 10 12:16:36.662608 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.662552    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c07be0\"/\"openshift-service-ca.crt\""
Sep 10 12:16:36.672085 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.672061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:16:36.760614 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.760614 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.760827 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.760827 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.760919 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6gw\" (UniqueName: \"kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.760919 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.760903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862428 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862829 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6gw\" (UniqueName: \"kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862890 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862890 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.862979 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.862922    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.864903 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.864881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.865119 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.865102    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.874532 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.874512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6gw\" (UniqueName: \"kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw\") pod \"rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") " pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:36.968547 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:36.968459    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:37.242579 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:37.242552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:16:37.244140 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:16:37.244111    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod567f3912_7b01_4275_a79d_ad83249f5a9e.slice/crio-84f3addca488b6ab94b674fbe35c739518b3661e59c31b6c4e5905f85e247709 WatchSource:0}: Error finding container 84f3addca488b6ab94b674fbe35c739518b3661e59c31b6c4e5905f85e247709: Status 404 returned error can't find the container with id 84f3addca488b6ab94b674fbe35c739518b3661e59c31b6c4e5905f85e247709
Sep 10 12:16:37.472610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:37.472571    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" event={"ID":"567f3912-7b01-4275-a79d-ad83249f5a9e","Type":"ContainerStarted","Data":"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"}
Sep 10 12:16:37.472610 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:37.472617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" event={"ID":"567f3912-7b01-4275-a79d-ad83249f5a9e","Type":"ContainerStarted","Data":"84f3addca488b6ab94b674fbe35c739518b3661e59c31b6c4e5905f85e247709"}
Sep 10 12:16:37.491098 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:37.491012    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" podStartSLOduration=1.490996985 podStartE2EDuration="1.490996985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:16:37.489906955 +0000 UTC m=+1499.069834377" watchObservedRunningTime="2026-09-10 12:16:37.490996985 +0000 UTC m=+1499.070924371"
Sep 10 12:16:39.007698 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:39.007668    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:16:39.010010 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:39.009990    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:16:46.100127 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.100088    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" podUID="eb504447-ed4c-4074-b685-89e49c821f77" containerName="main" containerID="cri-o://d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07" gracePeriod=45
Sep 10 12:16:46.348648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.348627    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:16:46.456268 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456240    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456344    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456364    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456403    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x89g8\" (UniqueName: \"kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456427    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456671 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456468    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home\") pod \"eb504447-ed4c-4074-b685-89e49c821f77\" (UID: \"eb504447-ed4c-4074-b685-89e49c821f77\") "
Sep 10 12:16:46.456671 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456514    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:16:46.456786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456692    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:16:46.456786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456772    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home" (OuterVolumeSpecName: "home") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:16:46.456898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456799    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.456898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.456816    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.458787 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.458766    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:16:46.458916 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.458802    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm" (OuterVolumeSpecName: "dshm") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:16:46.458916 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.458810    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8" (OuterVolumeSpecName: "kube-api-access-x89g8") pod "eb504447-ed4c-4074-b685-89e49c821f77" (UID: "eb504447-ed4c-4074-b685-89e49c821f77"). InnerVolumeSpecName "kube-api-access-x89g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:16:46.526418 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.526387    2573 generic.go:356] "Generic (PLEG): container finished" podID="eb504447-ed4c-4074-b685-89e49c821f77" containerID="d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07" exitCode=0
Sep 10 12:16:46.526555 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.526433    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"
Sep 10 12:16:46.526555 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.526468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" event={"ID":"eb504447-ed4c-4074-b685-89e49c821f77","Type":"ContainerDied","Data":"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"}
Sep 10 12:16:46.526555 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.526514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt" event={"ID":"eb504447-ed4c-4074-b685-89e49c821f77","Type":"ContainerDied","Data":"d947cfc0e342c7b999893bff4fc3324f6d26843361e3918e6e5fc7e30bc26378"}
Sep 10 12:16:46.526555 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.526536    2573 scope.go:117] "RemoveContainer" containerID="d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"
Sep 10 12:16:46.535689 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.535672    2573 scope.go:117] "RemoveContainer" containerID="d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"
Sep 10 12:16:46.535953 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:16:46.535932    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07\": container with ID starting with d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07 not found: ID does not exist" containerID="d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"
Sep 10 12:16:46.535994 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.535961    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07"} err="failed to get container status \"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07\": rpc error: code = NotFound desc = could not find container \"d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07\": container with ID starting with d919864449a27859701020d14ee75c9dc007404c1ffd878e271a16ad8c677d07 not found: ID does not exist"
Sep 10 12:16:46.547712 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.547691    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:16:46.553556 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.553537    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/sub-test-model-7ed5ac-kserve-6dcd86b665-bfbmt"]
Sep 10 12:16:46.557913 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.557895    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89g8\" (UniqueName: \"kubernetes.io/projected/eb504447-ed4c-4074-b685-89e49c821f77-kube-api-access-x89g8\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.557980 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.557916    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.557980 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.557926    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb504447-ed4c-4074-b685-89e49c821f77-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.557980 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.557933    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb504447-ed4c-4074-b685-89e49c821f77-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:46.946769 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.946737    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb504447-ed4c-4074-b685-89e49c821f77" path="/var/lib/kubelet/pods/eb504447-ed4c-4074-b685-89e49c821f77/volumes"
Sep 10 12:16:46.968709 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.968678    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:46.968887 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.968729    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:46.981730 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:46.981703    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:47.544320 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:47.544279    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:16:49.815680 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:49.815644    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:16:49.816093 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:49.815914    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" podUID="4f430ff2-9593-470c-85d9-048ca1f07801" containerName="payload-processing" containerID="cri-o://794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1" gracePeriod=30
Sep 10 12:16:50.071252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.071194    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:16:50.192541 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.192512    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbq8r\" (UniqueName: \"kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r\") pod \"4f430ff2-9593-470c-85d9-048ca1f07801\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") "
Sep 10 12:16:50.192710 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.192585    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume\") pod \"4f430ff2-9593-470c-85d9-048ca1f07801\" (UID: \"4f430ff2-9593-470c-85d9-048ca1f07801\") "
Sep 10 12:16:50.192986 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.192953    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f430ff2-9593-470c-85d9-048ca1f07801" (UID: "4f430ff2-9593-470c-85d9-048ca1f07801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:16:50.194760 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.194735    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r" (OuterVolumeSpecName: "kube-api-access-hbq8r") pod "4f430ff2-9593-470c-85d9-048ca1f07801" (UID: "4f430ff2-9593-470c-85d9-048ca1f07801"). InnerVolumeSpecName "kube-api-access-hbq8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:16:50.293461 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.293433    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbq8r\" (UniqueName: \"kubernetes.io/projected/4f430ff2-9593-470c-85d9-048ca1f07801-kube-api-access-hbq8r\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:50.293461 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.293457    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f430ff2-9593-470c-85d9-048ca1f07801-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:16:50.550465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.550432    2573 generic.go:356] "Generic (PLEG): container finished" podID="4f430ff2-9593-470c-85d9-048ca1f07801" containerID="794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1" exitCode=0
Sep 10 12:16:50.550645 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.550476    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"
Sep 10 12:16:50.550645 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.550514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" event={"ID":"4f430ff2-9593-470c-85d9-048ca1f07801","Type":"ContainerDied","Data":"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"}
Sep 10 12:16:50.550645 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.550567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6" event={"ID":"4f430ff2-9593-470c-85d9-048ca1f07801","Type":"ContainerDied","Data":"f04af2e3961e927fadb176ca8360c46bd0eb86090edc5f79c68b2cef987ac5b8"}
Sep 10 12:16:50.550645 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.550589    2573 scope.go:117] "RemoveContainer" containerID="794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"
Sep 10 12:16:50.560898 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.560881    2573 scope.go:117] "RemoveContainer" containerID="794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"
Sep 10 12:16:50.561176 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:16:50.561157    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1\": container with ID starting with 794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1 not found: ID does not exist" containerID="794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"
Sep 10 12:16:50.561252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.561184    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1"} err="failed to get container status \"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1\": rpc error: code = NotFound desc = could not find container \"794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1\": container with ID starting with 794e4ef753eef5eaa2cf857da01f01a8539d421837ee9d0707ea98e6bc5142d1 not found: ID does not exist"
Sep 10 12:16:50.583895 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.583873    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:16:50.601351 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.601322    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c531cb-bb594c859-792p6"]
Sep 10 12:16:50.948036 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:16:50.948008    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f430ff2-9593-470c-85d9-048ca1f07801" path="/var/lib/kubelet/pods/4f430ff2-9593-470c-85d9-048ca1f07801/volumes"
Sep 10 12:17:08.105998 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.105915    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:17:08.112779 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.112739    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:17:08.113029 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.113003    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" podUID="f26c0102-933c-4a71-8b92-eb8395702dc4" containerName="payload-pre-processing" containerID="cri-o://afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840" gracePeriod=30
Sep 10 12:17:08.369500 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.369438    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:17:08.466059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.466031    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume\") pod \"f26c0102-933c-4a71-8b92-eb8395702dc4\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") "
Sep 10 12:17:08.466225 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.466093    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pzdh\" (UniqueName: \"kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh\") pod \"f26c0102-933c-4a71-8b92-eb8395702dc4\" (UID: \"f26c0102-933c-4a71-8b92-eb8395702dc4\") "
Sep 10 12:17:08.466502 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.466479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f26c0102-933c-4a71-8b92-eb8395702dc4" (UID: "f26c0102-933c-4a71-8b92-eb8395702dc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:17:08.468341 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.468316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh" (OuterVolumeSpecName: "kube-api-access-7pzdh") pod "f26c0102-933c-4a71-8b92-eb8395702dc4" (UID: "f26c0102-933c-4a71-8b92-eb8395702dc4"). InnerVolumeSpecName "kube-api-access-7pzdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:17:08.567845 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.567814    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c0102-933c-4a71-8b92-eb8395702dc4-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:08.567845 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.567838    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pzdh\" (UniqueName: \"kubernetes.io/projected/f26c0102-933c-4a71-8b92-eb8395702dc4-kube-api-access-7pzdh\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:08.646167 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646133    2573 generic.go:356] "Generic (PLEG): container finished" podID="f26c0102-933c-4a71-8b92-eb8395702dc4" containerID="afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840" exitCode=0
Sep 10 12:17:08.646379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646181    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"
Sep 10 12:17:08.646379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" event={"ID":"f26c0102-933c-4a71-8b92-eb8395702dc4","Type":"ContainerDied","Data":"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"}
Sep 10 12:17:08.646379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646283    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl" event={"ID":"f26c0102-933c-4a71-8b92-eb8395702dc4","Type":"ContainerDied","Data":"5f1b2fd80f125be4f9cc5a21fc0f1f340108c53fa4243178fc65326ed06813f0"}
Sep 10 12:17:08.646379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646338    2573 scope.go:117] "RemoveContainer" containerID="afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"
Sep 10 12:17:08.646592 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.646544    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" podUID="5cc20928-b32c-4e29-998f-737fc3f08ce1" containerName="payload-processing" containerID="cri-o://b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8" gracePeriod=30
Sep 10 12:17:08.656390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.656371    2573 scope.go:117] "RemoveContainer" containerID="afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"
Sep 10 12:17:08.656707 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:17:08.656684    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840\": container with ID starting with afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840 not found: ID does not exist" containerID="afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"
Sep 10 12:17:08.656802 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.656718    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840"} err="failed to get container status \"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840\": rpc error: code = NotFound desc = could not find container \"afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840\": container with ID starting with afd6be1bc507424f2f52dda9b5d85ae4033a351f5227ad02b98a911a1e3fe840 not found: ID does not exist"
Sep 10 12:17:08.709852 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.709831    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:17:08.716907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.716886    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-639beb-6b66f6f766-tschl"]
Sep 10 12:17:08.882368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.882340    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:17:08.955406 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.955328    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f26c0102-933c-4a71-8b92-eb8395702dc4" path="/var/lib/kubelet/pods/f26c0102-933c-4a71-8b92-eb8395702dc4/volumes"
Sep 10 12:17:08.972555 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.972531    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdw4g\" (UniqueName: \"kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g\") pod \"5cc20928-b32c-4e29-998f-737fc3f08ce1\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") "
Sep 10 12:17:08.972678 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.972584    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume\") pod \"5cc20928-b32c-4e29-998f-737fc3f08ce1\" (UID: \"5cc20928-b32c-4e29-998f-737fc3f08ce1\") "
Sep 10 12:17:08.972974 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.972953    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cc20928-b32c-4e29-998f-737fc3f08ce1" (UID: "5cc20928-b32c-4e29-998f-737fc3f08ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:17:08.974761 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:08.974744    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g" (OuterVolumeSpecName: "kube-api-access-rdw4g") pod "5cc20928-b32c-4e29-998f-737fc3f08ce1" (UID: "5cc20928-b32c-4e29-998f-737fc3f08ce1"). InnerVolumeSpecName "kube-api-access-rdw4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:17:09.073569 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.073537    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdw4g\" (UniqueName: \"kubernetes.io/projected/5cc20928-b32c-4e29-998f-737fc3f08ce1-kube-api-access-rdw4g\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:09.073569 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.073564    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc20928-b32c-4e29-998f-737fc3f08ce1-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:09.653975 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.653941    2573 generic.go:356] "Generic (PLEG): container finished" podID="5cc20928-b32c-4e29-998f-737fc3f08ce1" containerID="b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8" exitCode=0
Sep 10 12:17:09.654453 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.653987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" event={"ID":"5cc20928-b32c-4e29-998f-737fc3f08ce1","Type":"ContainerDied","Data":"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"}
Sep 10 12:17:09.654453 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.654002    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"
Sep 10 12:17:09.654453 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.654038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l" event={"ID":"5cc20928-b32c-4e29-998f-737fc3f08ce1","Type":"ContainerDied","Data":"8e7a8afbafe4910c66818feec76610a8d8975bb2342f7e18a397d0e67284e217"}
Sep 10 12:17:09.654453 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.654068    2573 scope.go:117] "RemoveContainer" containerID="b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"
Sep 10 12:17:09.663024 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.663003    2573 scope.go:117] "RemoveContainer" containerID="b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"
Sep 10 12:17:09.663274 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:17:09.663257    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8\": container with ID starting with b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8 not found: ID does not exist" containerID="b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"
Sep 10 12:17:09.663338 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.663281    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8"} err="failed to get container status \"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8\": rpc error: code = NotFound desc = could not find container \"b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8\": container with ID starting with b594871280482d5f013ec713b208f2bdcc2061545a9d9e77c80b39147db6cfb8 not found: ID does not exist"
Sep 10 12:17:09.677491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.677468    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:17:09.682101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:09.682081    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-639beb-65977bdc7f-spt7l"]
Sep 10 12:17:10.948053 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:10.948017    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc20928-b32c-4e29-998f-737fc3f08ce1" path="/var/lib/kubelet/pods/5cc20928-b32c-4e29-998f-737fc3f08ce1/volumes"
Sep 10 12:17:18.498729 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.498691    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:17:18.499150 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499136    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f26c0102-933c-4a71-8b92-eb8395702dc4" containerName="payload-pre-processing"
Sep 10 12:17:18.499194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499152    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f26c0102-933c-4a71-8b92-eb8395702dc4" containerName="payload-pre-processing"
Sep 10 12:17:18.499194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499164    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb504447-ed4c-4074-b685-89e49c821f77" containerName="main"
Sep 10 12:17:18.499194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499171    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb504447-ed4c-4074-b685-89e49c821f77" containerName="main"
Sep 10 12:17:18.499194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499192    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc20928-b32c-4e29-998f-737fc3f08ce1" containerName="payload-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499197    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc20928-b32c-4e29-998f-737fc3f08ce1" containerName="payload-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499206    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f430ff2-9593-470c-85d9-048ca1f07801" containerName="payload-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499212    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f430ff2-9593-470c-85d9-048ca1f07801" containerName="payload-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499276    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc20928-b32c-4e29-998f-737fc3f08ce1" containerName="payload-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499282    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f26c0102-933c-4a71-8b92-eb8395702dc4" containerName="payload-pre-processing"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499291    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb504447-ed4c-4074-b685-89e49c821f77" containerName="main"
Sep 10 12:17:18.499340 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.499296    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f430ff2-9593-470c-85d9-048ca1f07801" containerName="payload-processing"
Sep 10 12:17:18.503660 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.503643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.507196 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.507174    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"openshift-service-ca.crt\""
Sep 10 12:17:18.508476 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.508456    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"kube-root-ca.crt\""
Sep 10 12:17:18.508594 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.508501    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"default-dockercfg-69mpl\""
Sep 10 12:17:18.508594 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.508501    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ed5ac\"/\"rate-test-model-7ed5ac-kserve-self-signed-certs\""
Sep 10 12:17:18.536392 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.536368    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:17:18.563022 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.562994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6ss\" (UniqueName: \"kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.563182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.563045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.563182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.563153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.563182 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.563176    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.563380 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.563205    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.563380 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.563257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664169 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664169 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6ss\" (UniqueName: \"kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664370    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664781 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664852 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.664893 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.664861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.666618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.666589    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.667009 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.666987    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.675235 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.675212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6ss\" (UniqueName: \"kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss\") pod \"rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") " pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:18.813130 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:18.813044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:19.094573 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:19.094542    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:17:19.095254 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:17:19.095221    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod554b64a7_6739_4f79_a441_542b9175e0ff.slice/crio-02ec52505c97ed857e05461f23ba6ba1c8488f180fc76601dad43cd37964294a WatchSource:0}: Error finding container 02ec52505c97ed857e05461f23ba6ba1c8488f180fc76601dad43cd37964294a: Status 404 returned error can't find the container with id 02ec52505c97ed857e05461f23ba6ba1c8488f180fc76601dad43cd37964294a
Sep 10 12:17:19.707497 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:19.707463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" event={"ID":"554b64a7-6739-4f79-a441-542b9175e0ff","Type":"ContainerStarted","Data":"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"}
Sep 10 12:17:19.707888 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:19.707512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" event={"ID":"554b64a7-6739-4f79-a441-542b9175e0ff","Type":"ContainerStarted","Data":"02ec52505c97ed857e05461f23ba6ba1c8488f180fc76601dad43cd37964294a"}
Sep 10 12:17:19.748388 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:19.748339    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" podStartSLOduration=1.748319258 podStartE2EDuration="1.748319258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:17:19.747392041 +0000 UTC m=+1541.327319427" watchObservedRunningTime="2026-09-10 12:17:19.748319258 +0000 UTC m=+1541.328246642"
Sep 10 12:17:25.996558 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:25.996523    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:17:25.997004 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:25.996730    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" podUID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" containerName="maas-api" containerID="cri-o://ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce" gracePeriod=30
Sep 10 12:17:26.084343 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.084292    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:17:26.084537 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.084517    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" podUID="42f6ee90-5df9-4116-8439-83e2abd5678a" containerName="payload-processing" containerID="cri-o://37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565" gracePeriod=30
Sep 10 12:17:26.337922 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.337893    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:17:26.364475 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.364453    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:17:26.438097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438067    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzpbm\" (UniqueName: \"kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm\") pod \"42f6ee90-5df9-4116-8439-83e2abd5678a\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") "
Sep 10 12:17:26.438252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438166    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtlzp\" (UniqueName: \"kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp\") pod \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") "
Sep 10 12:17:26.438252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs\") pod \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") "
Sep 10 12:17:26.438252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438204    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume\") pod \"42f6ee90-5df9-4116-8439-83e2abd5678a\" (UID: \"42f6ee90-5df9-4116-8439-83e2abd5678a\") "
Sep 10 12:17:26.438252 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438249    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca\") pod \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") "
Sep 10 12:17:26.438495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438341    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") pod \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\" (UID: \"aa5fe72a-9ca3-4d5f-90d1-e5559b709372\") "
Sep 10 12:17:26.438693 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42f6ee90-5df9-4116-8439-83e2abd5678a" (UID: "42f6ee90-5df9-4116-8439-83e2abd5678a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:17:26.438826 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.438742    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa5fe72a-9ca3-4d5f-90d1-e5559b709372" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:17:26.440713 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.440689    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa5fe72a-9ca3-4d5f-90d1-e5559b709372" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:17:26.441063 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.441044    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa5fe72a-9ca3-4d5f-90d1-e5559b709372" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:17:26.441146 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.441060    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm" (OuterVolumeSpecName: "kube-api-access-bzpbm") pod "42f6ee90-5df9-4116-8439-83e2abd5678a" (UID: "42f6ee90-5df9-4116-8439-83e2abd5678a"). InnerVolumeSpecName "kube-api-access-bzpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:17:26.441146 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.441072    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp" (OuterVolumeSpecName: "kube-api-access-vtlzp") pod "aa5fe72a-9ca3-4d5f-90d1-e5559b709372" (UID: "aa5fe72a-9ca3-4d5f-90d1-e5559b709372"). InnerVolumeSpecName "kube-api-access-vtlzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.539958    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlzp\" (UniqueName: \"kubernetes.io/projected/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-kube-api-access-vtlzp\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.539983    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.539993    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42f6ee90-5df9-4116-8439-83e2abd5678a-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.540003    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.540012    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa5fe72a-9ca3-4d5f-90d1-e5559b709372-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.540031 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.540021    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpbm\" (UniqueName: \"kubernetes.io/projected/42f6ee90-5df9-4116-8439-83e2abd5678a-kube-api-access-bzpbm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:17:26.745977 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.745940    2573 generic.go:356] "Generic (PLEG): container finished" podID="42f6ee90-5df9-4116-8439-83e2abd5678a" containerID="37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565" exitCode=0
Sep 10 12:17:26.746175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.745998    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"
Sep 10 12:17:26.746175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.746029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" event={"ID":"42f6ee90-5df9-4116-8439-83e2abd5678a","Type":"ContainerDied","Data":"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"}
Sep 10 12:17:26.746175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.746097    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw" event={"ID":"42f6ee90-5df9-4116-8439-83e2abd5678a","Type":"ContainerDied","Data":"3ae553d03edd9794e571b7c5eb5d789ae0ab2a4748ff251009b1192198627694"}
Sep 10 12:17:26.746175 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.746131    2573 scope.go:117] "RemoveContainer" containerID="37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"
Sep 10 12:17:26.747465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.747426    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" containerID="ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce" exitCode=0
Sep 10 12:17:26.747537 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.747464    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"
Sep 10 12:17:26.747537 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.747497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" event={"ID":"aa5fe72a-9ca3-4d5f-90d1-e5559b709372","Type":"ContainerDied","Data":"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"}
Sep 10 12:17:26.747613 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.747540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp" event={"ID":"aa5fe72a-9ca3-4d5f-90d1-e5559b709372","Type":"ContainerDied","Data":"751006489c73b5285e410dcfb58f11dbd7dae042bcf7980c9f4fe099c0176831"}
Sep 10 12:17:26.754944 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.754917    2573 scope.go:117] "RemoveContainer" containerID="37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"
Sep 10 12:17:26.755203 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:17:26.755180    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565\": container with ID starting with 37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565 not found: ID does not exist" containerID="37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"
Sep 10 12:17:26.755256 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.755214    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565"} err="failed to get container status \"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565\": rpc error: code = NotFound desc = could not find container \"37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565\": container with ID starting with 37df7b5332dd6fb02e72f61ffd93ca4c8d13ecf0f0112bd57abc2b221feb1565 not found: ID does not exist"
Sep 10 12:17:26.755256 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.755238    2573 scope.go:117] "RemoveContainer" containerID="ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"
Sep 10 12:17:26.763680 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.763665    2573 scope.go:117] "RemoveContainer" containerID="ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"
Sep 10 12:17:26.763972 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:17:26.763953    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce\": container with ID starting with ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce not found: ID does not exist" containerID="ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"
Sep 10 12:17:26.764048 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.763981    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce"} err="failed to get container status \"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce\": rpc error: code = NotFound desc = could not find container \"ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce\": container with ID starting with ed72bdcf242776935617e4c44feecc4082947a4eb5a91c169971421a27e67fce not found: ID does not exist"
Sep 10 12:17:26.786033 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.786001    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:17:26.795956 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.795906    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c66562-58fb9dfbd7-8j8vw"]
Sep 10 12:17:26.834998 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.834967    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:17:26.846399 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.846377    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c66562-5946dbc875-rnpmp"]
Sep 10 12:17:26.947428 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.947397    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f6ee90-5df9-4116-8439-83e2abd5678a" path="/var/lib/kubelet/pods/42f6ee90-5df9-4116-8439-83e2abd5678a/volumes"
Sep 10 12:17:26.947811 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:26.947798    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" path="/var/lib/kubelet/pods/aa5fe72a-9ca3-4d5f-90d1-e5559b709372/volumes"
Sep 10 12:17:28.813163 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:28.813120    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:28.813665 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:28.813205    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:28.826006 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:28.825987    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:17:29.776221 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:17:29.776193    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:18:23.459817 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:23.459784    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:18:24.657707 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:24.657662    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:18:31.012364 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012273    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:18:31.012767 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012742    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f6ee90-5df9-4116-8439-83e2abd5678a" containerName="payload-processing"
Sep 10 12:18:31.012767 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012767    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f6ee90-5df9-4116-8439-83e2abd5678a" containerName="payload-processing"
Sep 10 12:18:31.012856 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012787    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" containerName="maas-api"
Sep 10 12:18:31.012856 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012793    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" containerName="maas-api"
Sep 10 12:18:31.012919 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012859    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f6ee90-5df9-4116-8439-83e2abd5678a" containerName="payload-processing"
Sep 10 12:18:31.012919 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.012867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5fe72a-9ca3-4d5f-90d1-e5559b709372" containerName="maas-api"
Sep 10 12:18:31.015837 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.015820    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.018769 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.018749    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19f6c3\""
Sep 10 12:18:31.026814 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.026791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:18:31.175152 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.175115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrfd\" (UniqueName: \"kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.175343 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.175192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.276363 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.276250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrfd\" (UniqueName: \"kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.276363 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.276354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.277051 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.277032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.285619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.285600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrfd\" (UniqueName: \"kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd\") pod \"payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.327048 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.327018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:31.597291 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:31.597267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:18:31.599619 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:18:31.599585    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11802e5d_1d06_449a_b7d7_c6e3357d6f29.slice/crio-567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a WatchSource:0}: Error finding container 567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a: Status 404 returned error can't find the container with id 567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a
Sep 10 12:18:32.076686 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:32.076653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" event={"ID":"11802e5d-1d06-449a-b7d7-c6e3357d6f29","Type":"ContainerStarted","Data":"4c2dedb2347c178045d1a2e34d4c2f2d82c721480ab2d32c36ff3d3630656348"}
Sep 10 12:18:32.077086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:32.076712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" event={"ID":"11802e5d-1d06-449a-b7d7-c6e3357d6f29","Type":"ContainerStarted","Data":"567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a"}
Sep 10 12:18:32.077086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:32.076735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:32.095430 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:32.095372    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" podStartSLOduration=2.095355453 podStartE2EDuration="2.095355453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:18:32.093015729 +0000 UTC m=+1613.672943118" watchObservedRunningTime="2026-09-10 12:18:32.095355453 +0000 UTC m=+1613.675282838"
Sep 10 12:18:38.479805 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.479736    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" podUID="567f3912-7b01-4275-a79d-ad83249f5a9e" containerName="main" containerID="cri-o://f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8" gracePeriod=45
Sep 10 12:18:38.725660 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.725632    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:18:38.745441 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745382    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745441 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745441 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745489    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745621 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745592    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745619    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn6gw\" (UniqueName: \"kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw\") pod \"567f3912-7b01-4275-a79d-ad83249f5a9e\" (UID: \"567f3912-7b01-4275-a79d-ad83249f5a9e\") "
Sep 10 12:18:38.745736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745651    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home" (OuterVolumeSpecName: "home") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:38.745736 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745696    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:38.745895 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745824    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:38.745980 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745964    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:38.746036 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745986    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:38.746036 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.745999    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:38.747602 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.747585    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm" (OuterVolumeSpecName: "dshm") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:38.747772 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.747754    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:18:38.747868 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.747836    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw" (OuterVolumeSpecName: "kube-api-access-zn6gw") pod "567f3912-7b01-4275-a79d-ad83249f5a9e" (UID: "567f3912-7b01-4275-a79d-ad83249f5a9e"). InnerVolumeSpecName "kube-api-access-zn6gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:18:38.847081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.847033    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567f3912-7b01-4275-a79d-ad83249f5a9e-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:38.847081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.847073    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6gw\" (UniqueName: \"kubernetes.io/projected/567f3912-7b01-4275-a79d-ad83249f5a9e-kube-api-access-zn6gw\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:38.847081 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:38.847088    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567f3912-7b01-4275-a79d-ad83249f5a9e-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:39.111203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.111106    2573 generic.go:356] "Generic (PLEG): container finished" podID="567f3912-7b01-4275-a79d-ad83249f5a9e" containerID="f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8" exitCode=0
Sep 10 12:18:39.111203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.111163    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"
Sep 10 12:18:39.111203 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.111192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" event={"ID":"567f3912-7b01-4275-a79d-ad83249f5a9e","Type":"ContainerDied","Data":"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"}
Sep 10 12:18:39.111512 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.111248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm" event={"ID":"567f3912-7b01-4275-a79d-ad83249f5a9e","Type":"ContainerDied","Data":"84f3addca488b6ab94b674fbe35c739518b3661e59c31b6c4e5905f85e247709"}
Sep 10 12:18:39.111512 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.111270    2573 scope.go:117] "RemoveContainer" containerID="f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"
Sep 10 12:18:39.120097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.120078    2573 scope.go:117] "RemoveContainer" containerID="f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"
Sep 10 12:18:39.120396 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:18:39.120378    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8\": container with ID starting with f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8 not found: ID does not exist" containerID="f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"
Sep 10 12:18:39.120450 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.120406    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8"} err="failed to get container status \"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8\": rpc error: code = NotFound desc = could not find container \"f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8\": container with ID starting with f2378076284910ff360c3f5e24ecd1b8c7ad388b2fb2b24f73093b4bc5e64ba8 not found: ID does not exist"
Sep 10 12:18:39.127786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.127764    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:18:39.131495 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.131474    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c07be0/rate-test-model-c07be0-kserve-85fc6dd94c-qbtxm"]
Sep 10 12:18:39.681148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.681105    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" containerName="main" containerID="cri-o://5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe" gracePeriod=45
Sep 10 12:18:39.765501 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.765457    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 10 12:18:39.928591 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.928570    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:18:39.957954 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.957886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958102 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.957976    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958102 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958045    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958102 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958072    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958259 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958109    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958259 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958142    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw6ss\" (UniqueName: \"kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss\") pod \"554b64a7-6739-4f79-a441-542b9175e0ff\" (UID: \"554b64a7-6739-4f79-a441-542b9175e0ff\") "
Sep 10 12:18:39.958259 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958191    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home" (OuterVolumeSpecName: "home") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:39.958259 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958226    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:39.958525 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958360    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:39.958587 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958568    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:39.958625 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958592    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:39.958625 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.958608    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:39.960329 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.960274    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm" (OuterVolumeSpecName: "dshm") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:18:39.960752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.960726    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:18:39.960841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:39.960816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss" (OuterVolumeSpecName: "kube-api-access-jw6ss") pod "554b64a7-6739-4f79-a441-542b9175e0ff" (UID: "554b64a7-6739-4f79-a441-542b9175e0ff"). InnerVolumeSpecName "kube-api-access-jw6ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:18:40.059702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.059666    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/554b64a7-6739-4f79-a441-542b9175e0ff-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:40.059702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.059695    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/554b64a7-6739-4f79-a441-542b9175e0ff-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:40.059702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.059704    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6ss\" (UniqueName: \"kubernetes.io/projected/554b64a7-6739-4f79-a441-542b9175e0ff-kube-api-access-jw6ss\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:18:40.117097 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.117067    2573 generic.go:356] "Generic (PLEG): container finished" podID="554b64a7-6739-4f79-a441-542b9175e0ff" containerID="5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe" exitCode=0
Sep 10 12:18:40.117227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.117109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" event={"ID":"554b64a7-6739-4f79-a441-542b9175e0ff","Type":"ContainerDied","Data":"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"}
Sep 10 12:18:40.117227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.117114    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"
Sep 10 12:18:40.117227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.117136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp" event={"ID":"554b64a7-6739-4f79-a441-542b9175e0ff","Type":"ContainerDied","Data":"02ec52505c97ed857e05461f23ba6ba1c8488f180fc76601dad43cd37964294a"}
Sep 10 12:18:40.117227 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.117153    2573 scope.go:117] "RemoveContainer" containerID="5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"
Sep 10 12:18:40.126810 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.126791    2573 scope.go:117] "RemoveContainer" containerID="5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"
Sep 10 12:18:40.127081 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:18:40.127066    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe\": container with ID starting with 5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe not found: ID does not exist" containerID="5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"
Sep 10 12:18:40.127132 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.127088    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe"} err="failed to get container status \"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe\": rpc error: code = NotFound desc = could not find container \"5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe\": container with ID starting with 5ab4b858057a829df18c504e90ffb0e62a0d4671cefa8f502d2c07d217aa0cfe not found: ID does not exist"
Sep 10 12:18:40.139006 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.138980    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:18:40.143354 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.143331    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ed5ac/rate-test-model-7ed5ac-kserve-5497fc9fbc-zndzp"]
Sep 10 12:18:40.948171 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.948139    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" path="/var/lib/kubelet/pods/554b64a7-6739-4f79-a441-542b9175e0ff/volumes"
Sep 10 12:18:40.948571 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:40.948552    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567f3912-7b01-4275-a79d-ad83249f5a9e" path="/var/lib/kubelet/pods/567f3912-7b01-4275-a79d-ad83249f5a9e/volumes"
Sep 10 12:18:43.084480 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:43.084445    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:18:59.731183 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731144    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731598    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" containerName="main"
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731611    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" containerName="main"
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731632    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567f3912-7b01-4275-a79d-ad83249f5a9e" containerName="main"
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731638    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="567f3912-7b01-4275-a79d-ad83249f5a9e" containerName="main"
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731708    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="554b64a7-6739-4f79-a441-542b9175e0ff" containerName="main"
Sep 10 12:18:59.731877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.731717    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="567f3912-7b01-4275-a79d-ad83249f5a9e" containerName="main"
Sep 10 12:18:59.736544 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.736522    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.739741 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.739722    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-742017-serving-cert\""
Sep 10 12:18:59.752687 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.752666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:18:59.841968 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.841937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.842138 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.842099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.842138 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.842129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.842238 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.842146    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj929\" (UniqueName: \"kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.942901 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.942861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.943101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.942911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.943101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.942939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj929\" (UniqueName: \"kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.943101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.942986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.943101 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:18:59.943024    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-serving-cert: secret "maas-api-e2e-ipp-b-742017-serving-cert" not found
Sep 10 12:18:59.943325 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:18:59.943112    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls podName:d6e178dc-2858-4be3-bb87-7eea96d2d238 nodeName:}" failed. No retries permitted until 2026-09-10 12:19:00.443090325 +0000 UTC m=+1642.023017692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls") pod "maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" (UID: "d6e178dc-2858-4be3-bb87-7eea96d2d238") : secret "maas-api-e2e-ipp-b-742017-serving-cert" not found
Sep 10 12:18:59.943795 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.943775    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.945458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.945436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:18:59.952926 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:18:59.952897    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj929\" (UniqueName: \"kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:00.396931 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.396895    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:19:00.400545 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.400520    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.403264 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.403234    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-742017-dockercfg-rltqp\""
Sep 10 12:19:00.403382 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.403359    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-742017\""
Sep 10 12:19:00.415400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.415378    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:19:00.421679 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.421655    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:19:00.425863 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.425844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.436625 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.436499    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:19:00.449206 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.449185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:00.452052 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.451883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") pod \"maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:00.550316 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.550273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.550454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.550334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8w4\" (UniqueName: \"kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.550454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.550400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.550454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.550433    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zgh\" (UniqueName: \"kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.648015 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.647938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:00.651237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.650819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zgh\" (UniqueName: \"kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.651237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.650873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.651237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.650911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8w4\" (UniqueName: \"kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.651237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.650954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.651630 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.651611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.651785 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.651765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.659345 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.659327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8w4\" (UniqueName: \"kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4\") pod \"payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.659432 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.659390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zgh\" (UniqueName: \"kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh\") pod \"payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.711694 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.711664    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:00.737864 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.737805    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:00.927429 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.927406    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:19:00.928971 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:19:00.928947    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e178dc_2858_4be3_bb87_7eea96d2d238.slice/crio-dc6868febd69d2841b6d0dcb4fdc0f917972fada41470ea812c4791c1d77eab6 WatchSource:0}: Error finding container dc6868febd69d2841b6d0dcb4fdc0f917972fada41470ea812c4791c1d77eab6: Status 404 returned error can't find the container with id dc6868febd69d2841b6d0dcb4fdc0f917972fada41470ea812c4791c1d77eab6
Sep 10 12:19:00.995915 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:00.995891    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:19:00.997992 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:19:00.997946    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20251c00_d942_45a2_8818_6ec79270dfe9.slice/crio-de4be549ea803079faa58a3802c55edd7733cc3acf98e96fdbf7340a6353a5b5 WatchSource:0}: Error finding container de4be549ea803079faa58a3802c55edd7733cc3acf98e96fdbf7340a6353a5b5: Status 404 returned error can't find the container with id de4be549ea803079faa58a3802c55edd7733cc3acf98e96fdbf7340a6353a5b5
Sep 10 12:19:01.028679 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.028660    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:19:01.035437 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:19:01.035413    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b648b23_c762_4700_a1fa_477621a4f7b5.slice/crio-ff49b1af673fa740c988280f608a3e3926e3c307feccb95ae1589e2fdc17922d WatchSource:0}: Error finding container ff49b1af673fa740c988280f608a3e3926e3c307feccb95ae1589e2fdc17922d: Status 404 returned error can't find the container with id ff49b1af673fa740c988280f608a3e3926e3c307feccb95ae1589e2fdc17922d
Sep 10 12:19:01.223763 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.223666    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" event={"ID":"7b648b23-c762-4700-a1fa-477621a4f7b5","Type":"ContainerStarted","Data":"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"}
Sep 10 12:19:01.223763 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.223723    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" event={"ID":"7b648b23-c762-4700-a1fa-477621a4f7b5","Type":"ContainerStarted","Data":"ff49b1af673fa740c988280f608a3e3926e3c307feccb95ae1589e2fdc17922d"}
Sep 10 12:19:01.223984 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.223809    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:01.225237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.225210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" event={"ID":"d6e178dc-2858-4be3-bb87-7eea96d2d238","Type":"ContainerStarted","Data":"dc6868febd69d2841b6d0dcb4fdc0f917972fada41470ea812c4791c1d77eab6"}
Sep 10 12:19:01.226914 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.226863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" event={"ID":"20251c00-d942-45a2-8818-6ec79270dfe9","Type":"ContainerStarted","Data":"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"}
Sep 10 12:19:01.226914 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.226912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" event={"ID":"20251c00-d942-45a2-8818-6ec79270dfe9","Type":"ContainerStarted","Data":"de4be549ea803079faa58a3802c55edd7733cc3acf98e96fdbf7340a6353a5b5"}
Sep 10 12:19:01.227094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.227023    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:01.241535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.241487    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" podStartSLOduration=1.241471108 podStartE2EDuration="1.241471108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:19:01.240833419 +0000 UTC m=+1642.820760806" watchObservedRunningTime="2026-09-10 12:19:01.241471108 +0000 UTC m=+1642.821398495"
Sep 10 12:19:01.258978 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:01.258932    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" podStartSLOduration=1.258914903 podStartE2EDuration="1.258914903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:19:01.258278709 +0000 UTC m=+1642.838206105" watchObservedRunningTime="2026-09-10 12:19:01.258914903 +0000 UTC m=+1642.838842292"
Sep 10 12:19:02.233360 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:02.233327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" event={"ID":"d6e178dc-2858-4be3-bb87-7eea96d2d238","Type":"ContainerStarted","Data":"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"}
Sep 10 12:19:02.233766 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:02.233497    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:02.251513 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:02.251458    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" podStartSLOduration=2.887604237 podStartE2EDuration="3.251440759s" totalImagesPullingTime="363.836522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:19:02.251067844 +0000 UTC m=+1643.830995240" watchObservedRunningTime="2026-09-10 12:19:02.251440759 +0000 UTC m=+1643.831368147"
Sep 10 12:19:08.244529 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:08.244500    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:19:12.236078 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:12.236046    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:19:12.236586 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:12.236118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:19:28.903458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:28.903431    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7_7b648b23-c762-4700-a1fa-477621a4f7b5/payload-processing/0.log"
Sep 10 12:19:37.055366 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.055333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:19:37.060408 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.060385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.064284 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.064259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-19f6c3\"/\"default-dockercfg-p8245\""
Sep 10 12:19:37.064424 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.064287    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-19f6c3\"/\"openshift-service-ca.crt\""
Sep 10 12:19:37.064424 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.064259    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-19f6c3\"/\"ipp-route-19f6c3-kserve-self-signed-certs\""
Sep 10 12:19:37.064627 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.064610    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-19f6c3\"/\"kube-root-ca.crt\""
Sep 10 12:19:37.068148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.068125    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:19:37.203001 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.202958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.203001 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.203005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.203241 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.203041    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrhd\" (UniqueName: \"kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.203241 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.203104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.203241 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.203154    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.203241 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.203233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.303886 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.303839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304035 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.303923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304035 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.303958    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304035 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrhd\" (UniqueName: \"kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304194 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304441 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.304535 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.304489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.306420 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.306366    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.306543 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.306524    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.312701 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.312683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrhd\" (UniqueName: \"kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd\") pod \"ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") " pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.372909 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.372886    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:37.645962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:37.645935    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:19:37.647802 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:19:37.647766    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24b85e71_7eab_4a65_81f1_ddb936a69ba2.slice/crio-da9dbc0756627acd5c76ce9870fac01f15dd0c87bd07897ad42b1969cd9cd23a WatchSource:0}: Error finding container da9dbc0756627acd5c76ce9870fac01f15dd0c87bd07897ad42b1969cd9cd23a: Status 404 returned error can't find the container with id da9dbc0756627acd5c76ce9870fac01f15dd0c87bd07897ad42b1969cd9cd23a
Sep 10 12:19:38.419768 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:38.419726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" event={"ID":"24b85e71-7eab-4a65-81f1-ddb936a69ba2","Type":"ContainerStarted","Data":"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"}
Sep 10 12:19:38.420148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:38.419778    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" event={"ID":"24b85e71-7eab-4a65-81f1-ddb936a69ba2","Type":"ContainerStarted","Data":"da9dbc0756627acd5c76ce9870fac01f15dd0c87bd07897ad42b1969cd9cd23a"}
Sep 10 12:19:38.437230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:38.437178    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" podStartSLOduration=1.437164332 podStartE2EDuration="1.437164332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:19:38.43615965 +0000 UTC m=+1680.016087037" watchObservedRunningTime="2026-09-10 12:19:38.437164332 +0000 UTC m=+1680.017091718"
Sep 10 12:19:47.373385 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:47.373345    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:47.373855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:47.373397    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:47.386973 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:47.386945    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:19:47.479497 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:19:47.479472    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:20:09.955723 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:09.955650    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7_7b648b23-c762-4700-a1fa-477621a4f7b5/payload-processing/0.log"
Sep 10 12:20:15.965219 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:15.965179    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:15.968452 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:15.968435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:15.972687 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:15.972665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2fc5d5-dockercfg-c29dx\""
Sep 10 12:20:15.972827 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:15.972712    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2fc5d5\""
Sep 10 12:20:15.994711 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:15.994688    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:16.065751 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.065720    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:16.069435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.069413    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.076762 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.076625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wlf\" (UniqueName: \"kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.076762 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.076694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.079501 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.079475    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:16.178291 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.178256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wlf\" (UniqueName: \"kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.178525 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.178346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.178525 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.178406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.178525 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.178446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwvq\" (UniqueName: \"kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.179054 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.179034    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.187677 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.187650    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wlf\" (UniqueName: \"kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf\") pod \"payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.278933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.278831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:16.279114 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.279093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.279164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.279137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwvq\" (UniqueName: \"kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.279777 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.279758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.287404 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.287383    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwvq\" (UniqueName: \"kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq\") pod \"payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.382065 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.382024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:16.553124 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.553101    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:16.554746 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:20:16.554725    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f75f50_38fc_45f3_bc07_ce514922d1e9.slice/crio-8cdc37083d5d6baa6d55f6de86dd66d6d6aaaf3eb05c17879ca67cf1565fa4ff WatchSource:0}: Error finding container 8cdc37083d5d6baa6d55f6de86dd66d6d6aaaf3eb05c17879ca67cf1565fa4ff: Status 404 returned error can't find the container with id 8cdc37083d5d6baa6d55f6de86dd66d6d6aaaf3eb05c17879ca67cf1565fa4ff
Sep 10 12:20:16.613750 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.613721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" event={"ID":"a9f75f50-38fc-45f3-bc07-ce514922d1e9","Type":"ContainerStarted","Data":"8cdc37083d5d6baa6d55f6de86dd66d6d6aaaf3eb05c17879ca67cf1565fa4ff"}
Sep 10 12:20:16.662281 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:16.662248    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:16.663429 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:20:16.663395    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2192e0_2ff6_40d2_9dab_f896a185b04f.slice/crio-4446527725dd6faf50445e2751990b307e9ae3968c8d764b06a9e94b267a7802 WatchSource:0}: Error finding container 4446527725dd6faf50445e2751990b307e9ae3968c8d764b06a9e94b267a7802: Status 404 returned error can't find the container with id 4446527725dd6faf50445e2751990b307e9ae3968c8d764b06a9e94b267a7802
Sep 10 12:20:17.622436 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.622325    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" event={"ID":"5b2192e0-2ff6-40d2-9dab-f896a185b04f","Type":"ContainerStarted","Data":"28982d51153f21b098b835f443f34cd477e4470429b6b7518c7862b24d21229d"}
Sep 10 12:20:17.622882 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.622440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" event={"ID":"5b2192e0-2ff6-40d2-9dab-f896a185b04f","Type":"ContainerStarted","Data":"4446527725dd6faf50445e2751990b307e9ae3968c8d764b06a9e94b267a7802"}
Sep 10 12:20:17.622882 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.622472    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:17.624059 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.624031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" event={"ID":"a9f75f50-38fc-45f3-bc07-ce514922d1e9","Type":"ContainerStarted","Data":"2be69d4693b866d0ca1e022cd1e9d64f5c8d908061835af898020cb854b3bd5f"}
Sep 10 12:20:17.624212 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.624081    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:17.642619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.642567    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" podStartSLOduration=1.642548143 podStartE2EDuration="1.642548143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:20:17.639171972 +0000 UTC m=+1719.219099370" watchObservedRunningTime="2026-09-10 12:20:17.642548143 +0000 UTC m=+1719.222475535"
Sep 10 12:20:17.658274 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:17.658224    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" podStartSLOduration=2.658206685 podStartE2EDuration="2.658206685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:20:17.656123215 +0000 UTC m=+1719.236050603" watchObservedRunningTime="2026-09-10 12:20:17.658206685 +0000 UTC m=+1719.238134073"
Sep 10 12:20:28.631167 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:28.631139    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:28.631592 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:28.631201    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:45.607115 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.607083    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:45.607675 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.607291    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" podUID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" containerName="payload-processing" containerID="cri-o://28982d51153f21b098b835f443f34cd477e4470429b6b7518c7862b24d21229d" gracePeriod=30
Sep 10 12:20:45.613468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.613259    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:45.613614 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.613565    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" podUID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" containerName="payload-pre-processing" containerID="cri-o://2be69d4693b866d0ca1e022cd1e9d64f5c8d908061835af898020cb854b3bd5f" gracePeriod=30
Sep 10 12:20:45.775649 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.775609    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" containerID="28982d51153f21b098b835f443f34cd477e4470429b6b7518c7862b24d21229d" exitCode=0
Sep 10 12:20:45.775649 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.775634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" event={"ID":"5b2192e0-2ff6-40d2-9dab-f896a185b04f","Type":"ContainerDied","Data":"28982d51153f21b098b835f443f34cd477e4470429b6b7518c7862b24d21229d"}
Sep 10 12:20:45.777705 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.777664    2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" containerID="2be69d4693b866d0ca1e022cd1e9d64f5c8d908061835af898020cb854b3bd5f" exitCode=0
Sep 10 12:20:45.777847 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.777696    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" event={"ID":"a9f75f50-38fc-45f3-bc07-ce514922d1e9","Type":"ContainerDied","Data":"2be69d4693b866d0ca1e022cd1e9d64f5c8d908061835af898020cb854b3bd5f"}
Sep 10 12:20:45.870390 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.870282    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:45.896096 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.896070    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:45.977491 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.977454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghwvq\" (UniqueName: \"kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq\") pod \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") "
Sep 10 12:20:45.977664 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.977509    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2wlf\" (UniqueName: \"kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf\") pod \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") "
Sep 10 12:20:45.977664 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.977538    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume\") pod \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\" (UID: \"a9f75f50-38fc-45f3-bc07-ce514922d1e9\") "
Sep 10 12:20:45.977664 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.977635    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume\") pod \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\" (UID: \"5b2192e0-2ff6-40d2-9dab-f896a185b04f\") "
Sep 10 12:20:45.977957 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.977928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9f75f50-38fc-45f3-bc07-ce514922d1e9" (UID: "a9f75f50-38fc-45f3-bc07-ce514922d1e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:20:45.978086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.978023    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b2192e0-2ff6-40d2-9dab-f896a185b04f" (UID: "5b2192e0-2ff6-40d2-9dab-f896a185b04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:20:45.979720 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.979702    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf" (OuterVolumeSpecName: "kube-api-access-n2wlf") pod "a9f75f50-38fc-45f3-bc07-ce514922d1e9" (UID: "a9f75f50-38fc-45f3-bc07-ce514922d1e9"). InnerVolumeSpecName "kube-api-access-n2wlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:20:45.979792 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:45.979751    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq" (OuterVolumeSpecName: "kube-api-access-ghwvq") pod "5b2192e0-2ff6-40d2-9dab-f896a185b04f" (UID: "5b2192e0-2ff6-40d2-9dab-f896a185b04f"). InnerVolumeSpecName "kube-api-access-ghwvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:20:46.079174 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.079132    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwvq\" (UniqueName: \"kubernetes.io/projected/5b2192e0-2ff6-40d2-9dab-f896a185b04f-kube-api-access-ghwvq\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:20:46.079174 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.079162    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wlf\" (UniqueName: \"kubernetes.io/projected/a9f75f50-38fc-45f3-bc07-ce514922d1e9-kube-api-access-n2wlf\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:20:46.079174 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.079171    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f75f50-38fc-45f3-bc07-ce514922d1e9-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:20:46.079174 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.079179    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2192e0-2ff6-40d2-9dab-f896a185b04f-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:20:46.785254 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.785227    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"
Sep 10 12:20:46.785855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.785226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc" event={"ID":"5b2192e0-2ff6-40d2-9dab-f896a185b04f","Type":"ContainerDied","Data":"4446527725dd6faf50445e2751990b307e9ae3968c8d764b06a9e94b267a7802"}
Sep 10 12:20:46.785855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.785421    2573 scope.go:117] "RemoveContainer" containerID="28982d51153f21b098b835f443f34cd477e4470429b6b7518c7862b24d21229d"
Sep 10 12:20:46.786964 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.786820    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"
Sep 10 12:20:46.786964 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.786870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v" event={"ID":"a9f75f50-38fc-45f3-bc07-ce514922d1e9","Type":"ContainerDied","Data":"8cdc37083d5d6baa6d55f6de86dd66d6d6aaaf3eb05c17879ca67cf1565fa4ff"}
Sep 10 12:20:46.795585 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.795568    2573 scope.go:117] "RemoveContainer" containerID="2be69d4693b866d0ca1e022cd1e9d64f5c8d908061835af898020cb854b3bd5f"
Sep 10 12:20:46.811811 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.811791    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:46.826164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.826142    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2fc5d5-5ddd7987bf-pd8bc"]
Sep 10 12:20:46.849550 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.849529    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:46.861058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.861039    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2fc5d5-5df4767798-n5v6v"]
Sep 10 12:20:46.948052 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.948025    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" path="/var/lib/kubelet/pods/5b2192e0-2ff6-40d2-9dab-f896a185b04f/volumes"
Sep 10 12:20:46.948424 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:20:46.948411    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" path="/var/lib/kubelet/pods/a9f75f50-38fc-45f3-bc07-ce514922d1e9/volumes"
Sep 10 12:21:11.662464 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.662431    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:21:11.662943 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.662691    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" podUID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" containerName="payload-pre-processing" containerID="cri-o://4c2dedb2347c178045d1a2e34d4c2f2d82c721480ab2d32c36ff3d3630656348" gracePeriod=30
Sep 10 12:21:11.927690 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.927596    2573 generic.go:356] "Generic (PLEG): container finished" podID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" containerID="4c2dedb2347c178045d1a2e34d4c2f2d82c721480ab2d32c36ff3d3630656348" exitCode=0
Sep 10 12:21:11.928017 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.927682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" event={"ID":"11802e5d-1d06-449a-b7d7-c6e3357d6f29","Type":"ContainerDied","Data":"4c2dedb2347c178045d1a2e34d4c2f2d82c721480ab2d32c36ff3d3630656348"}
Sep 10 12:21:11.928209 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.928192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q" event={"ID":"11802e5d-1d06-449a-b7d7-c6e3357d6f29","Type":"ContainerDied","Data":"567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a"}
Sep 10 12:21:11.928351 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.928337    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="567e497e37bc506c8a86e7c7a8bb8758d9bf7aa084142245abb4052b42c8115a"
Sep 10 12:21:11.938663 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:11.938637    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:21:12.033876 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.033832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxrfd\" (UniqueName: \"kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd\") pod \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") "
Sep 10 12:21:12.033876 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.033876    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume\") pod \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\" (UID: \"11802e5d-1d06-449a-b7d7-c6e3357d6f29\") "
Sep 10 12:21:12.034271 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.034241    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11802e5d-1d06-449a-b7d7-c6e3357d6f29" (UID: "11802e5d-1d06-449a-b7d7-c6e3357d6f29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:21:12.036101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.036076    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd" (OuterVolumeSpecName: "kube-api-access-mxrfd") pod "11802e5d-1d06-449a-b7d7-c6e3357d6f29" (UID: "11802e5d-1d06-449a-b7d7-c6e3357d6f29"). InnerVolumeSpecName "kube-api-access-mxrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:21:12.135465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.135428    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxrfd\" (UniqueName: \"kubernetes.io/projected/11802e5d-1d06-449a-b7d7-c6e3357d6f29-kube-api-access-mxrfd\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:12.135465 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.135458    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11802e5d-1d06-449a-b7d7-c6e3357d6f29-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:12.933623 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.933592    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"
Sep 10 12:21:12.958508 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.958477    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:21:12.962918 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:12.962891    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-19f6c3-57c87984f8-6gg5q"]
Sep 10 12:21:14.946488 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:14.946446    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" path="/var/lib/kubelet/pods/11802e5d-1d06-449a-b7d7-c6e3357d6f29/volumes"
Sep 10 12:21:17.229522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:17.229488    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:21:32.253351 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.253229    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" podUID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" containerName="main" containerID="cri-o://954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded" gracePeriod=45
Sep 10 12:21:32.501485 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.501458    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:21:32.638786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638692    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.638786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638750    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.638786 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638784    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.639073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638809    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.639073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.639073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qrhd\" (UniqueName: \"kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd\") pod \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\" (UID: \"24b85e71-7eab-4a65-81f1-ddb936a69ba2\") "
Sep 10 12:21:32.639073 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.638994    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:21:32.639264 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.639098    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:21:32.639264 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.639118    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home" (OuterVolumeSpecName: "home") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:21:32.639391 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.639330    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:32.639391 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.639348    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:32.639391 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.639357    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:32.641189 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.641163    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:21:32.641293 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.641204    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd" (OuterVolumeSpecName: "kube-api-access-6qrhd") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "kube-api-access-6qrhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:21:32.641293 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.641237    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm" (OuterVolumeSpecName: "dshm") pod "24b85e71-7eab-4a65-81f1-ddb936a69ba2" (UID: "24b85e71-7eab-4a65-81f1-ddb936a69ba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:21:32.740675 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.740639    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b85e71-7eab-4a65-81f1-ddb936a69ba2-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:32.740675 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.740669    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrhd\" (UniqueName: \"kubernetes.io/projected/24b85e71-7eab-4a65-81f1-ddb936a69ba2-kube-api-access-6qrhd\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:32.740675 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:32.740680    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b85e71-7eab-4a65-81f1-ddb936a69ba2-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:21:33.037881 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.037850    2573 generic.go:356] "Generic (PLEG): container finished" podID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" containerID="954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded" exitCode=0
Sep 10 12:21:33.038058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.037897    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"
Sep 10 12:21:33.038058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.037936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" event={"ID":"24b85e71-7eab-4a65-81f1-ddb936a69ba2","Type":"ContainerDied","Data":"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"}
Sep 10 12:21:33.038058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.037980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w" event={"ID":"24b85e71-7eab-4a65-81f1-ddb936a69ba2","Type":"ContainerDied","Data":"da9dbc0756627acd5c76ce9870fac01f15dd0c87bd07897ad42b1969cd9cd23a"}
Sep 10 12:21:33.038058 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.037996    2573 scope.go:117] "RemoveContainer" containerID="954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"
Sep 10 12:21:33.047637 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.047617    2573 scope.go:117] "RemoveContainer" containerID="954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"
Sep 10 12:21:33.047950 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:21:33.047931    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded\": container with ID starting with 954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded not found: ID does not exist" containerID="954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"
Sep 10 12:21:33.048000 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.047963    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded"} err="failed to get container status \"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded\": rpc error: code = NotFound desc = could not find container \"954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded\": container with ID starting with 954a5569c75aae2a9a28dde0af52ef63dca972fbe091839a71b9e7d994999ded not found: ID does not exist"
Sep 10 12:21:33.058358 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.058322    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:21:33.065659 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:33.065636    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-19f6c3/ipp-route-19f6c3-kserve-7bdd749d4c-ssk4w"]
Sep 10 12:21:34.953047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:34.953013    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" path="/var/lib/kubelet/pods/24b85e71-7eab-4a65-81f1-ddb936a69ba2/volumes"
Sep 10 12:21:39.041644 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:39.041619    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:21:39.048672 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:21:39.048650    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:22:11.217703 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.217665    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:22:11.218223 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.217922    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" podUID="d6e178dc-2858-4be3-bb87-7eea96d2d238" containerName="maas-api" containerID="cri-o://26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57" gracePeriod=30
Sep 10 12:22:11.369953 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.369813    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:22:11.370139 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.370067    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" podUID="7b648b23-c762-4700-a1fa-477621a4f7b5" containerName="payload-processing" containerID="cri-o://69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc" gracePeriod=30
Sep 10 12:22:11.374404 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.374374    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:22:11.374674 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.374631    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" podUID="20251c00-d942-45a2-8818-6ec79270dfe9" containerName="payload-pre-processing" containerID="cri-o://6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a" gracePeriod=30
Sep 10 12:22:11.511819 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.511790    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:22:11.605200 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.605166    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj929\" (UniqueName: \"kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929\") pod \"d6e178dc-2858-4be3-bb87-7eea96d2d238\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") "
Sep 10 12:22:11.605400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.605235    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") pod \"d6e178dc-2858-4be3-bb87-7eea96d2d238\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") "
Sep 10 12:22:11.605400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.605266    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs\") pod \"d6e178dc-2858-4be3-bb87-7eea96d2d238\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") "
Sep 10 12:22:11.605400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.605339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca\") pod \"d6e178dc-2858-4be3-bb87-7eea96d2d238\" (UID: \"d6e178dc-2858-4be3-bb87-7eea96d2d238\") "
Sep 10 12:22:11.606006 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.605920    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6e178dc-2858-4be3-bb87-7eea96d2d238" (UID: "d6e178dc-2858-4be3-bb87-7eea96d2d238"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:11.608655 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.608628    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929" (OuterVolumeSpecName: "kube-api-access-xj929") pod "d6e178dc-2858-4be3-bb87-7eea96d2d238" (UID: "d6e178dc-2858-4be3-bb87-7eea96d2d238"). InnerVolumeSpecName "kube-api-access-xj929". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:11.610648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.610623    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6e178dc-2858-4be3-bb87-7eea96d2d238" (UID: "d6e178dc-2858-4be3-bb87-7eea96d2d238"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:11.610815 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.610794    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6e178dc-2858-4be3-bb87-7eea96d2d238" (UID: "d6e178dc-2858-4be3-bb87-7eea96d2d238"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:11.637171 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.637151    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:22:11.642455 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.642436    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:22:11.706752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.706717    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj929\" (UniqueName: \"kubernetes.io/projected/d6e178dc-2858-4be3-bb87-7eea96d2d238-kube-api-access-xj929\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.706752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.706746    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.706752 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.706757    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6e178dc-2858-4be3-bb87-7eea96d2d238-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.707067 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.706765    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e178dc-2858-4be3-bb87-7eea96d2d238-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.807637 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.807559    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume\") pod \"7b648b23-c762-4700-a1fa-477621a4f7b5\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") "
Sep 10 12:22:11.807637 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.807618    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs8w4\" (UniqueName: \"kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4\") pod \"20251c00-d942-45a2-8818-6ec79270dfe9\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") "
Sep 10 12:22:11.807808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.807663    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume\") pod \"20251c00-d942-45a2-8818-6ec79270dfe9\" (UID: \"20251c00-d942-45a2-8818-6ec79270dfe9\") "
Sep 10 12:22:11.807808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.807683    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9zgh\" (UniqueName: \"kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh\") pod \"7b648b23-c762-4700-a1fa-477621a4f7b5\" (UID: \"7b648b23-c762-4700-a1fa-477621a4f7b5\") "
Sep 10 12:22:11.808047 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.808017    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b648b23-c762-4700-a1fa-477621a4f7b5" (UID: "7b648b23-c762-4700-a1fa-477621a4f7b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:11.808164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.808135    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20251c00-d942-45a2-8818-6ec79270dfe9" (UID: "20251c00-d942-45a2-8818-6ec79270dfe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:11.809808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.809784    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4" (OuterVolumeSpecName: "kube-api-access-hs8w4") pod "20251c00-d942-45a2-8818-6ec79270dfe9" (UID: "20251c00-d942-45a2-8818-6ec79270dfe9"). InnerVolumeSpecName "kube-api-access-hs8w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:11.809903 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.809841    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh" (OuterVolumeSpecName: "kube-api-access-l9zgh") pod "7b648b23-c762-4700-a1fa-477621a4f7b5" (UID: "7b648b23-c762-4700-a1fa-477621a4f7b5"). InnerVolumeSpecName "kube-api-access-l9zgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:11.909368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.909341    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b648b23-c762-4700-a1fa-477621a4f7b5-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.909368 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.909367    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8w4\" (UniqueName: \"kubernetes.io/projected/20251c00-d942-45a2-8818-6ec79270dfe9-kube-api-access-hs8w4\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.909551 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.909376    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20251c00-d942-45a2-8818-6ec79270dfe9-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:11.909551 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:11.909385    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zgh\" (UniqueName: \"kubernetes.io/projected/7b648b23-c762-4700-a1fa-477621a4f7b5-kube-api-access-l9zgh\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:12.250560 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.250524    2573 generic.go:356] "Generic (PLEG): container finished" podID="20251c00-d942-45a2-8818-6ec79270dfe9" containerID="6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a" exitCode=0
Sep 10 12:22:12.250962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.250590    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"
Sep 10 12:22:12.250962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.250612    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" event={"ID":"20251c00-d942-45a2-8818-6ec79270dfe9","Type":"ContainerDied","Data":"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"}
Sep 10 12:22:12.250962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.250678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz" event={"ID":"20251c00-d942-45a2-8818-6ec79270dfe9","Type":"ContainerDied","Data":"de4be549ea803079faa58a3802c55edd7733cc3acf98e96fdbf7340a6353a5b5"}
Sep 10 12:22:12.250962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.250713    2573 scope.go:117] "RemoveContainer" containerID="6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"
Sep 10 12:22:12.252085 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.251938    2573 generic.go:356] "Generic (PLEG): container finished" podID="7b648b23-c762-4700-a1fa-477621a4f7b5" containerID="69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc" exitCode=0
Sep 10 12:22:12.252085 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.251991    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"
Sep 10 12:22:12.252085 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.252026    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" event={"ID":"7b648b23-c762-4700-a1fa-477621a4f7b5","Type":"ContainerDied","Data":"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"}
Sep 10 12:22:12.252085 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.252073    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7" event={"ID":"7b648b23-c762-4700-a1fa-477621a4f7b5","Type":"ContainerDied","Data":"ff49b1af673fa740c988280f608a3e3926e3c307feccb95ae1589e2fdc17922d"}
Sep 10 12:22:12.253779 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.253726    2573 generic.go:356] "Generic (PLEG): container finished" podID="d6e178dc-2858-4be3-bb87-7eea96d2d238" containerID="26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57" exitCode=0
Sep 10 12:22:12.253905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.253763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" event={"ID":"d6e178dc-2858-4be3-bb87-7eea96d2d238","Type":"ContainerDied","Data":"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"}
Sep 10 12:22:12.253905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.253796    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"
Sep 10 12:22:12.253905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.253824    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl" event={"ID":"d6e178dc-2858-4be3-bb87-7eea96d2d238","Type":"ContainerDied","Data":"dc6868febd69d2841b6d0dcb4fdc0f917972fada41470ea812c4791c1d77eab6"}
Sep 10 12:22:12.260514 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.260399    2573 scope.go:117] "RemoveContainer" containerID="6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"
Sep 10 12:22:12.260720 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:12.260693    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a\": container with ID starting with 6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a not found: ID does not exist" containerID="6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"
Sep 10 12:22:12.260814 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.260729    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a"} err="failed to get container status \"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a\": rpc error: code = NotFound desc = could not find container \"6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a\": container with ID starting with 6431e13f1e6414eba29ba23c7798f694149508c6b7577ac5e96a9b7670eea67a not found: ID does not exist"
Sep 10 12:22:12.260814 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.260752    2573 scope.go:117] "RemoveContainer" containerID="69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"
Sep 10 12:22:12.269929 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.269909    2573 scope.go:117] "RemoveContainer" containerID="69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"
Sep 10 12:22:12.270176 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:12.270158    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc\": container with ID starting with 69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc not found: ID does not exist" containerID="69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"
Sep 10 12:22:12.270239 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.270180    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc"} err="failed to get container status \"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc\": rpc error: code = NotFound desc = could not find container \"69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc\": container with ID starting with 69773bb87761eef19b40715c0ec1db776e4d0265213b77d927fe3d9f79e01efc not found: ID does not exist"
Sep 10 12:22:12.270239 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.270194    2573 scope.go:117] "RemoveContainer" containerID="26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"
Sep 10 12:22:12.277962 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.277945    2573 scope.go:117] "RemoveContainer" containerID="26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"
Sep 10 12:22:12.278205 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:12.278189    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57\": container with ID starting with 26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57 not found: ID does not exist" containerID="26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"
Sep 10 12:22:12.278266 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.278209    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57"} err="failed to get container status \"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57\": rpc error: code = NotFound desc = could not find container \"26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57\": container with ID starting with 26438e52f3a28acecf97a419e7a1880cb4eac08a4b2c75bce810e4ceb9f7cc57 not found: ID does not exist"
Sep 10 12:22:12.284629 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.284608    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:22:12.286470 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.286452    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-742017-94fd57b6-5lqgz"]
Sep 10 12:22:12.303351 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.303326    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:22:12.310211 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.310189    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-742017-7ccbc887b-ftjpl"]
Sep 10 12:22:12.328381 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.328363    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:22:12.331983 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.331962    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-742017-58fb7bffb9-h6nx7"]
Sep 10 12:22:12.947406 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.947376    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20251c00-d942-45a2-8818-6ec79270dfe9" path="/var/lib/kubelet/pods/20251c00-d942-45a2-8818-6ec79270dfe9/volumes"
Sep 10 12:22:12.947756 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.947743    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b648b23-c762-4700-a1fa-477621a4f7b5" path="/var/lib/kubelet/pods/7b648b23-c762-4700-a1fa-477621a4f7b5/volumes"
Sep 10 12:22:12.948090 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:12.948078    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6e178dc-2858-4be3-bb87-7eea96d2d238" path="/var/lib/kubelet/pods/d6e178dc-2858-4be3-bb87-7eea96d2d238/volumes"
Sep 10 12:22:30.540167 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.540137    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:22:30.540655 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.540418    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" podUID="f6ac4800-1316-432c-9a21-94c17661665e" containerName="maas-api" containerID="cri-o://f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299" gracePeriod=30
Sep 10 12:22:30.642289 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.642251    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:22:30.642558 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.642525    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" podUID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" containerName="payload-processing" containerID="cri-o://658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a" gracePeriod=30
Sep 10 12:22:30.818270 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.818241    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:22:30.883703 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.883670    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs\") pod \"f6ac4800-1316-432c-9a21-94c17661665e\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") "
Sep 10 12:22:30.883855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.883760    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca\") pod \"f6ac4800-1316-432c-9a21-94c17661665e\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") "
Sep 10 12:22:30.883855 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.883808    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p67kx\" (UniqueName: \"kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx\") pod \"f6ac4800-1316-432c-9a21-94c17661665e\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") "
Sep 10 12:22:30.883933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.883863    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") pod \"f6ac4800-1316-432c-9a21-94c17661665e\" (UID: \"f6ac4800-1316-432c-9a21-94c17661665e\") "
Sep 10 12:22:30.884135 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.884105    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6ac4800-1316-432c-9a21-94c17661665e" (UID: "f6ac4800-1316-432c-9a21-94c17661665e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:30.884232 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.884193    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ac4800-1316-432c-9a21-94c17661665e-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:30.887481 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.887445    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f6ac4800-1316-432c-9a21-94c17661665e" (UID: "f6ac4800-1316-432c-9a21-94c17661665e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:30.887481 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.887458    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6ac4800-1316-432c-9a21-94c17661665e" (UID: "f6ac4800-1316-432c-9a21-94c17661665e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:30.887669 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.887496    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx" (OuterVolumeSpecName: "kube-api-access-p67kx") pod "f6ac4800-1316-432c-9a21-94c17661665e" (UID: "f6ac4800-1316-432c-9a21-94c17661665e"). InnerVolumeSpecName "kube-api-access-p67kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:30.912865 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.912845    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:22:30.985132 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985108    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9djq\" (UniqueName: \"kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq\") pod \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") "
Sep 10 12:22:30.985257 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985162    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume\") pod \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\" (UID: \"2ee6ab99-583e-4faf-9abe-09959e5fbb34\") "
Sep 10 12:22:30.985489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985467    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67kx\" (UniqueName: \"kubernetes.io/projected/f6ac4800-1316-432c-9a21-94c17661665e-kube-api-access-p67kx\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:30.985489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985488    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:30.985665 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985502    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6ac4800-1316-432c-9a21-94c17661665e-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:30.985665 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.985529    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ee6ab99-583e-4faf-9abe-09959e5fbb34" (UID: "2ee6ab99-583e-4faf-9abe-09959e5fbb34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:30.987371 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:30.987352    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq" (OuterVolumeSpecName: "kube-api-access-z9djq") pod "2ee6ab99-583e-4faf-9abe-09959e5fbb34" (UID: "2ee6ab99-583e-4faf-9abe-09959e5fbb34"). InnerVolumeSpecName "kube-api-access-z9djq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:31.086458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.086393    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9djq\" (UniqueName: \"kubernetes.io/projected/2ee6ab99-583e-4faf-9abe-09959e5fbb34-kube-api-access-z9djq\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:31.086458 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.086416    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ee6ab99-583e-4faf-9abe-09959e5fbb34-plugins-config-volume\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:31.349755 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.349673    2573 generic.go:356] "Generic (PLEG): container finished" podID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" containerID="658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a" exitCode=0
Sep 10 12:22:31.349755 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.349724    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"
Sep 10 12:22:31.349967 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.349751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" event={"ID":"2ee6ab99-583e-4faf-9abe-09959e5fbb34","Type":"ContainerDied","Data":"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"}
Sep 10 12:22:31.349967 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.349811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js" event={"ID":"2ee6ab99-583e-4faf-9abe-09959e5fbb34","Type":"ContainerDied","Data":"ce05b893085aceaa63a94c27608fa9fdcd66b9346442d424572218ffbd847c75"}
Sep 10 12:22:31.349967 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.349830    2573 scope.go:117] "RemoveContainer" containerID="658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"
Sep 10 12:22:31.351255 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.351130    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6ac4800-1316-432c-9a21-94c17661665e" containerID="f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299" exitCode=0
Sep 10 12:22:31.351255 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.351190    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" event={"ID":"f6ac4800-1316-432c-9a21-94c17661665e","Type":"ContainerDied","Data":"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"}
Sep 10 12:22:31.351255 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.351219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp" event={"ID":"f6ac4800-1316-432c-9a21-94c17661665e","Type":"ContainerDied","Data":"7f0017198fec2df6bc1aac0663ead837cd8f471c7269e9609e5c6c7e154552b8"}
Sep 10 12:22:31.351443 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.351272    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"
Sep 10 12:22:31.359108 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.359091    2573 scope.go:117] "RemoveContainer" containerID="658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"
Sep 10 12:22:31.359385 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:31.359366    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a\": container with ID starting with 658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a not found: ID does not exist" containerID="658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"
Sep 10 12:22:31.359466 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.359394    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a"} err="failed to get container status \"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a\": rpc error: code = NotFound desc = could not find container \"658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a\": container with ID starting with 658f93ebc361a3ffe8de74c3eea0d0e92e6986c96e507524a642b6235d9e295a not found: ID does not exist"
Sep 10 12:22:31.359466 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.359411    2573 scope.go:117] "RemoveContainer" containerID="f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"
Sep 10 12:22:31.367002 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.366986    2573 scope.go:117] "RemoveContainer" containerID="f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"
Sep 10 12:22:31.367248 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:31.367231    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299\": container with ID starting with f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299 not found: ID does not exist" containerID="f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"
Sep 10 12:22:31.367295 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.367255    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299"} err="failed to get container status \"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299\": rpc error: code = NotFound desc = could not find container \"f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299\": container with ID starting with f3213eed82cfd3ac62854ccfffe18d9f84d037feb0fba702b117ec40dc23e299 not found: ID does not exist"
Sep 10 12:22:31.384816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.384785    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:22:31.389947 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.389926    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-c07be0-867d5fb9bb-rgkvp"]
Sep 10 12:22:31.418226 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.418194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:22:31.422615 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:31.422594    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c07be0-5db6c8c7f7-4n5js"]
Sep 10 12:22:32.946554 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:32.946524    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" path="/var/lib/kubelet/pods/2ee6ab99-583e-4faf-9abe-09959e5fbb34/volumes"
Sep 10 12:22:32.946926 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:32.946917    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ac4800-1316-432c-9a21-94c17661665e" path="/var/lib/kubelet/pods/f6ac4800-1316-432c-9a21-94c17661665e/volumes"
Sep 10 12:22:49.531973 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.531928    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:22:49.532567 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.532215    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" podUID="65154c05-b8ac-4be1-9f60-da804745deb3" containerName="maas-api" containerID="cri-o://190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1" gracePeriod=30
Sep 10 12:22:49.782869 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.782811    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:22:49.964229 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs\") pod \"65154c05-b8ac-4be1-9f60-da804745deb3\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") "
Sep 10 12:22:49.964229 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") pod \"65154c05-b8ac-4be1-9f60-da804745deb3\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") "
Sep 10 12:22:49.964499 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964263    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca\") pod \"65154c05-b8ac-4be1-9f60-da804745deb3\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") "
Sep 10 12:22:49.964499 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964348    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x24m\" (UniqueName: \"kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m\") pod \"65154c05-b8ac-4be1-9f60-da804745deb3\" (UID: \"65154c05-b8ac-4be1-9f60-da804745deb3\") "
Sep 10 12:22:49.964663 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964635    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65154c05-b8ac-4be1-9f60-da804745deb3" (UID: "65154c05-b8ac-4be1-9f60-da804745deb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:49.964823 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.964702    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65154c05-b8ac-4be1-9f60-da804745deb3-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:49.966487 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.966460    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65154c05-b8ac-4be1-9f60-da804745deb3" (UID: "65154c05-b8ac-4be1-9f60-da804745deb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:49.966625 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.966602    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m" (OuterVolumeSpecName: "kube-api-access-9x24m") pod "65154c05-b8ac-4be1-9f60-da804745deb3" (UID: "65154c05-b8ac-4be1-9f60-da804745deb3"). InnerVolumeSpecName "kube-api-access-9x24m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:49.966872 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:49.966854    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65154c05-b8ac-4be1-9f60-da804745deb3" (UID: "65154c05-b8ac-4be1-9f60-da804745deb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:22:50.065787 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.065702    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:50.065941 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.065797    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65154c05-b8ac-4be1-9f60-da804745deb3-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:50.065941 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.065807    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x24m\" (UniqueName: \"kubernetes.io/projected/65154c05-b8ac-4be1-9f60-da804745deb3-kube-api-access-9x24m\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:22:50.451296 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.451262    2573 generic.go:356] "Generic (PLEG): container finished" podID="65154c05-b8ac-4be1-9f60-da804745deb3" containerID="190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1" exitCode=0
Sep 10 12:22:50.451489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.451340    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"
Sep 10 12:22:50.451489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.451356    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" event={"ID":"65154c05-b8ac-4be1-9f60-da804745deb3","Type":"ContainerDied","Data":"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"}
Sep 10 12:22:50.451489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.451411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6" event={"ID":"65154c05-b8ac-4be1-9f60-da804745deb3","Type":"ContainerDied","Data":"347e1f5cd565808aa0ab65cacdb2e39cd9905262e395deb309b5cdce93989084"}
Sep 10 12:22:50.451489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.451435    2573 scope.go:117] "RemoveContainer" containerID="190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"
Sep 10 12:22:50.460976 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.460959    2573 scope.go:117] "RemoveContainer" containerID="190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"
Sep 10 12:22:50.461243 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:22:50.461223    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1\": container with ID starting with 190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1 not found: ID does not exist" containerID="190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"
Sep 10 12:22:50.461339 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.461252    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1"} err="failed to get container status \"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1\": rpc error: code = NotFound desc = could not find container \"190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1\": container with ID starting with 190fd595f7618ce99d3a2f97b747707ed6992427695454f8c66df9916b48f6b1 not found: ID does not exist"
Sep 10 12:22:50.492354 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.492327    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:22:50.500665 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.500644    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7ed5ac-5cb9d465df-rqcv6"]
Sep 10 12:22:50.947618 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:22:50.947586    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65154c05-b8ac-4be1-9f60-da804745deb3" path="/var/lib/kubelet/pods/65154c05-b8ac-4be1-9f60-da804745deb3/volumes"
Sep 10 12:24:39.282907 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:24:39.282834    2573 scope.go:117] "RemoveContainer" containerID="4c2dedb2347c178045d1a2e34d4c2f2d82c721480ab2d32c36ff3d3630656348"
Sep 10 12:26:39.074049 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:26:39.074023    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:26:39.081635 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:26:39.081611    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:27:32.272724 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.272642    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:27:32.273239 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273216    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ac4800-1316-432c-9a21-94c17661665e" containerName="maas-api"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273244    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ac4800-1316-432c-9a21-94c17661665e" containerName="maas-api"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273258    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65154c05-b8ac-4be1-9f60-da804745deb3" containerName="maas-api"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273268    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65154c05-b8ac-4be1-9f60-da804745deb3" containerName="maas-api"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273284    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" containerName="payload-processing"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273293    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" containerName="payload-processing"
Sep 10 12:27:32.273334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273328    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" containerName="main"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273338    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" containerName="main"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273352    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273361    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273376    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20251c00-d942-45a2-8818-6ec79270dfe9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273384    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20251c00-d942-45a2-8818-6ec79270dfe9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273411    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273420    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273435    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b648b23-c762-4700-a1fa-477621a4f7b5" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273443    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b648b23-c762-4700-a1fa-477621a4f7b5" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273453    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273461    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273475    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6e178dc-2858-4be3-bb87-7eea96d2d238" containerName="maas-api"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273484    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6e178dc-2858-4be3-bb87-7eea96d2d238" containerName="maas-api"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273576    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b85e71-7eab-4a65-81f1-ddb936a69ba2" containerName="main"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273590    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b648b23-c762-4700-a1fa-477621a4f7b5" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273598    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f75f50-38fc-45f3-bc07-ce514922d1e9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273608    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b2192e0-2ff6-40d2-9dab-f896a185b04f" containerName="payload-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273619    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="11802e5d-1d06-449a-b7d7-c6e3357d6f29" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273629    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ac4800-1316-432c-9a21-94c17661665e" containerName="maas-api"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273638    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6e178dc-2858-4be3-bb87-7eea96d2d238" containerName="maas-api"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273647    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65154c05-b8ac-4be1-9f60-da804745deb3" containerName="maas-api"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273653    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20251c00-d942-45a2-8818-6ec79270dfe9" containerName="payload-pre-processing"
Sep 10 12:27:32.273653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.273663    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee6ab99-583e-4faf-9abe-09959e5fbb34" containerName="payload-processing"
Sep 10 12:27:32.276948 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.276925    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.280538 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.280521    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4b778336-kserve-self-signed-certs\""
Sep 10 12:27:32.290161 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.290136    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:27:32.395148 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.395258 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpdg\" (UniqueName: \"kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.395258 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395190    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.395365 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.395365 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395350    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.395435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.395386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.495933 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.495907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.495954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.495980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496056 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpdg\" (UniqueName: \"kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496230 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496445 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496419    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496538 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.496538 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.496517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.498379 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.498360    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.498623 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.498606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.511101 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.511082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpdg\" (UniqueName: \"kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg\") pod \"e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") " pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.586126 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.586061    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:32.861808 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:32.861736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:27:32.864420 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:27:32.864392    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ece2fe4_dfeb_4630_9137_1693d2439a42.slice/crio-dd4ae3b60acf4524351677d0cf8c3e03ead37ccb0748e6b2d69cf7a4d50a0867 WatchSource:0}: Error finding container dd4ae3b60acf4524351677d0cf8c3e03ead37ccb0748e6b2d69cf7a4d50a0867: Status 404 returned error can't find the container with id dd4ae3b60acf4524351677d0cf8c3e03ead37ccb0748e6b2d69cf7a4d50a0867
Sep 10 12:27:33.814239 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:33.814198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" event={"ID":"1ece2fe4-dfeb-4630-9137-1693d2439a42","Type":"ContainerStarted","Data":"4ebe0dd5763b5136a84243ca84712ba59999ffb2b2dc1a362ef3458871c8516f"}
Sep 10 12:27:33.814239 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:33.814243    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" event={"ID":"1ece2fe4-dfeb-4630-9137-1693d2439a42","Type":"ContainerStarted","Data":"dd4ae3b60acf4524351677d0cf8c3e03ead37ccb0748e6b2d69cf7a4d50a0867"}
Sep 10 12:27:33.835159 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:33.835110    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" podStartSLOduration=1.83509485 podStartE2EDuration="1.83509485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:27:33.833695432 +0000 UTC m=+2155.413622819" watchObservedRunningTime="2026-09-10 12:27:33.83509485 +0000 UTC m=+2155.415022236"
Sep 10 12:27:42.586571 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:42.586538    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:42.586972 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:42.586585    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:42.599180 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:42.599157    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:27:42.870086 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:27:42.870004    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:28:00.850546 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:00.850511    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:28:15.870648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:15.870586    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" podUID="1ece2fe4-dfeb-4630-9137-1693d2439a42" containerName="main" containerID="cri-o://4ebe0dd5763b5136a84243ca84712ba59999ffb2b2dc1a362ef3458871c8516f" gracePeriod=45
Sep 10 12:28:16.019867 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.019832    2573 generic.go:356] "Generic (PLEG): container finished" podID="1ece2fe4-dfeb-4630-9137-1693d2439a42" containerID="4ebe0dd5763b5136a84243ca84712ba59999ffb2b2dc1a362ef3458871c8516f" exitCode=0
Sep 10 12:28:16.020078 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.019906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" event={"ID":"1ece2fe4-dfeb-4630-9137-1693d2439a42","Type":"ContainerDied","Data":"4ebe0dd5763b5136a84243ca84712ba59999ffb2b2dc1a362ef3458871c8516f"}
Sep 10 12:28:16.122144 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.122082    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:28:16.303600 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303570    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tpdg\" (UniqueName: \"kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.303816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303637    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.303816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.303816 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303700    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.303993 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303875    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.303993 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.303945    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home\") pod \"1ece2fe4-dfeb-4630-9137-1693d2439a42\" (UID: \"1ece2fe4-dfeb-4630-9137-1693d2439a42\") "
Sep 10 12:28:16.304152 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304050    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:28:16.304234 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304206    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:28:16.304292 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304257    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home" (OuterVolumeSpecName: "home") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:28:16.304513 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304441    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-model-cache\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:16.304513 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304466    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-tmp-dir\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:16.304513 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.304480    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-home\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:16.306591 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.306558    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm" (OuterVolumeSpecName: "dshm") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:28:16.306591 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.306563    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:28:16.306770 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.306640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg" (OuterVolumeSpecName: "kube-api-access-5tpdg") pod "1ece2fe4-dfeb-4630-9137-1693d2439a42" (UID: "1ece2fe4-dfeb-4630-9137-1693d2439a42"). InnerVolumeSpecName "kube-api-access-5tpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:28:16.405164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.405121    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tpdg\" (UniqueName: \"kubernetes.io/projected/1ece2fe4-dfeb-4630-9137-1693d2439a42-kube-api-access-5tpdg\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:16.405164 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.405159    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ece2fe4-dfeb-4630-9137-1693d2439a42-tls-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:16.405421 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:16.405173    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ece2fe4-dfeb-4630-9137-1693d2439a42-dshm\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:17.026435 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:17.026400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm" event={"ID":"1ece2fe4-dfeb-4630-9137-1693d2439a42","Type":"ContainerDied","Data":"dd4ae3b60acf4524351677d0cf8c3e03ead37ccb0748e6b2d69cf7a4d50a0867"}
Sep 10 12:28:17.026877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:17.026447    2573 scope.go:117] "RemoveContainer" containerID="4ebe0dd5763b5136a84243ca84712ba59999ffb2b2dc1a362ef3458871c8516f"
Sep 10 12:28:17.026877 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:17.026414    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"
Sep 10 12:28:17.049029 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:17.048993    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:28:17.052846 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:17.052816    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4b778336-kserve-5c8d9c9567-6ndgm"]
Sep 10 12:28:18.947857 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:18.947827    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ece2fe4-dfeb-4630-9137-1693d2439a42" path="/var/lib/kubelet/pods/1ece2fe4-dfeb-4630-9137-1693d2439a42/volumes"
Sep 10 12:28:20.377702 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.377665    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:20.378237 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.378217    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ece2fe4-dfeb-4630-9137-1693d2439a42" containerName="main"
Sep 10 12:28:20.378338 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.378240    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ece2fe4-dfeb-4630-9137-1693d2439a42" containerName="main"
Sep 10 12:28:20.378400 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.378362    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ece2fe4-dfeb-4630-9137-1693d2439a42" containerName="main"
Sep 10 12:28:20.381514 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.381492    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.384425 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.384400    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:28:20.385494 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.385469    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 12:28:20.385604 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.385532    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjsm9\""
Sep 10 12:28:20.401048 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.401022    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:20.442331 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.442275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.442522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.442341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqv4\" (UniqueName: \"kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.442522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.442463    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.442522 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.442491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.543276 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.543234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.543485 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.543290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqv4\" (UniqueName: \"kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.543485 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.543451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.543485 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.543478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.544136 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.544115    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.545874 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.545854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.546017 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.545994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.560915 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.560893    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqv4\" (UniqueName: \"kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4\") pod \"maas-api-5bb5f968cb-6g8hl\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") " pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.692159 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.692129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:20.966492 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.966468    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:20.968594 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:28:20.968571    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04bc52d_e225_4c52_ad2b_e1297f64be8f.slice/crio-52b6a2a202bfd46ae17fc7198eb7f1926c0dafdc423a5cdb11fdb7f2b88d94f9 WatchSource:0}: Error finding container 52b6a2a202bfd46ae17fc7198eb7f1926c0dafdc423a5cdb11fdb7f2b88d94f9: Status 404 returned error can't find the container with id 52b6a2a202bfd46ae17fc7198eb7f1926c0dafdc423a5cdb11fdb7f2b88d94f9
Sep 10 12:28:20.969993 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:20.969974    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:28:21.051765 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:21.051730    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" event={"ID":"c04bc52d-e225-4c52-ad2b-e1297f64be8f","Type":"ContainerStarted","Data":"52b6a2a202bfd46ae17fc7198eb7f1926c0dafdc423a5cdb11fdb7f2b88d94f9"}
Sep 10 12:28:22.058832 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:22.058798    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" event={"ID":"c04bc52d-e225-4c52-ad2b-e1297f64be8f","Type":"ContainerStarted","Data":"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"}
Sep 10 12:28:22.059236 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:22.058915    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:22.080192 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:22.080139    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" podStartSLOduration=1.670526672 podStartE2EDuration="2.08012337s" totalImagesPullingTime="409.596698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:22.078467069 +0000 UTC m=+2203.658394454" watchObservedRunningTime="2026-09-10 12:28:22.08012337 +0000 UTC m=+2203.660050864"
Sep 10 12:28:28.069599 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.069563    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:28.243076 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.243041    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"]
Sep 10 12:28:28.246718 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.246701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.257361 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.257335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"]
Sep 10 12:28:28.313103 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.313068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-maas-api-tls\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.313103 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.313104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb089bf-4dfe-4561-b718-9311743ea944-openshift-service-ca\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.313334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.313185    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-metrics-certs\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.313334 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.313204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqkv\" (UniqueName: \"kubernetes.io/projected/ebb089bf-4dfe-4561-b718-9311743ea944-kube-api-access-lvqkv\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.414278 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.414249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-maas-api-tls\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.414468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.414285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb089bf-4dfe-4561-b718-9311743ea944-openshift-service-ca\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.414468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.414340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-metrics-certs\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.414468 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.414356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqkv\" (UniqueName: \"kubernetes.io/projected/ebb089bf-4dfe-4561-b718-9311743ea944-kube-api-access-lvqkv\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.415084 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.415060    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb089bf-4dfe-4561-b718-9311743ea944-openshift-service-ca\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.416880 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.416856    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-maas-api-tls\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.417001 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.416944    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb089bf-4dfe-4561-b718-9311743ea944-metrics-certs\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.424516 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.424489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqkv\" (UniqueName: \"kubernetes.io/projected/ebb089bf-4dfe-4561-b718-9311743ea944-kube-api-access-lvqkv\") pod \"maas-api-645cf68f76-d58lt\" (UID: \"ebb089bf-4dfe-4561-b718-9311743ea944\") " pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.558955 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.558922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:28.839449 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:28.839366    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"]
Sep 10 12:28:28.842657 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:28:28.842632    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb089bf_4dfe_4561_b718_9311743ea944.slice/crio-4f7ad1713517c7a640815ac7af0c700a6b58e0bc4c08e6ab6621b9dad2bbb804 WatchSource:0}: Error finding container 4f7ad1713517c7a640815ac7af0c700a6b58e0bc4c08e6ab6621b9dad2bbb804: Status 404 returned error can't find the container with id 4f7ad1713517c7a640815ac7af0c700a6b58e0bc4c08e6ab6621b9dad2bbb804
Sep 10 12:28:29.094975 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:29.094885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt" event={"ID":"ebb089bf-4dfe-4561-b718-9311743ea944","Type":"ContainerStarted","Data":"4f7ad1713517c7a640815ac7af0c700a6b58e0bc4c08e6ab6621b9dad2bbb804"}
Sep 10 12:28:30.101653 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:30.101617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt" event={"ID":"ebb089bf-4dfe-4561-b718-9311743ea944","Type":"ContainerStarted","Data":"2db746feaeb92ca423670ed9a529ac655587927b590e9a23726959a7b63b5cfd"}
Sep 10 12:28:30.102111 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:30.101755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:30.122636 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:30.122578    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt" podStartSLOduration=1.609542762 podStartE2EDuration="2.122562173s" totalImagesPullingTime="513.019411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:30.119644826 +0000 UTC m=+2211.699572212" watchObservedRunningTime="2026-09-10 12:28:30.122562173 +0000 UTC m=+2211.702489609"
Sep 10 12:28:36.113001 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.112970    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-645cf68f76-d58lt"
Sep 10 12:28:36.195246 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.195217    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:36.195437 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.195415    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" podUID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" containerName="maas-api" containerID="cri-o://542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b" gracePeriod=30
Sep 10 12:28:36.446620 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.446597    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:36.490242 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.490201    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtqv4\" (UniqueName: \"kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4\") pod \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") "
Sep 10 12:28:36.490454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.490274    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls\") pod \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") "
Sep 10 12:28:36.490454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.490319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs\") pod \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") "
Sep 10 12:28:36.490454 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.490384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca\") pod \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\" (UID: \"c04bc52d-e225-4c52-ad2b-e1297f64be8f\") "
Sep 10 12:28:36.490737 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.490702    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c04bc52d-e225-4c52-ad2b-e1297f64be8f" (UID: "c04bc52d-e225-4c52-ad2b-e1297f64be8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:28:36.492542 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.492514    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c04bc52d-e225-4c52-ad2b-e1297f64be8f" (UID: "c04bc52d-e225-4c52-ad2b-e1297f64be8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:28:36.492669 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.492612    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c04bc52d-e225-4c52-ad2b-e1297f64be8f" (UID: "c04bc52d-e225-4c52-ad2b-e1297f64be8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:28:36.492669 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.492637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4" (OuterVolumeSpecName: "kube-api-access-rtqv4") pod "c04bc52d-e225-4c52-ad2b-e1297f64be8f" (UID: "c04bc52d-e225-4c52-ad2b-e1297f64be8f"). InnerVolumeSpecName "kube-api-access-rtqv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:28:36.591371 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.591337    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqv4\" (UniqueName: \"kubernetes.io/projected/c04bc52d-e225-4c52-ad2b-e1297f64be8f-kube-api-access-rtqv4\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:36.591371 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.591366    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-maas-api-tls\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:36.591371 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.591376    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04bc52d-e225-4c52-ad2b-e1297f64be8f-metrics-certs\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:36.591600 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:36.591385    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c04bc52d-e225-4c52-ad2b-e1297f64be8f-openshift-service-ca\") on node \"ip-10-0-134-58.ec2.internal\" DevicePath \"\""
Sep 10 12:28:37.137489 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.137450    2573 generic.go:356] "Generic (PLEG): container finished" podID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" containerID="542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b" exitCode=0
Sep 10 12:28:37.137905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.137498    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"
Sep 10 12:28:37.137905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.137498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" event={"ID":"c04bc52d-e225-4c52-ad2b-e1297f64be8f","Type":"ContainerDied","Data":"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"}
Sep 10 12:28:37.137905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.137618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl" event={"ID":"c04bc52d-e225-4c52-ad2b-e1297f64be8f","Type":"ContainerDied","Data":"52b6a2a202bfd46ae17fc7198eb7f1926c0dafdc423a5cdb11fdb7f2b88d94f9"}
Sep 10 12:28:37.137905 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.137647    2573 scope.go:117] "RemoveContainer" containerID="542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"
Sep 10 12:28:37.146715 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.146690    2573 scope.go:117] "RemoveContainer" containerID="542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"
Sep 10 12:28:37.146985 ip-10-0-134-58 kubenswrapper[2573]: E0910 12:28:37.146964    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b\": container with ID starting with 542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b not found: ID does not exist" containerID="542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"
Sep 10 12:28:37.147057 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.146997    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b"} err="failed to get container status \"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b\": rpc error: code = NotFound desc = could not find container \"542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b\": container with ID starting with 542083304f47b364cea6d27b954b538f039d2e27b564a2043245db7a37cfb87b not found: ID does not exist"
Sep 10 12:28:37.166195 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.166157    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:37.174593 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:37.174568    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bb5f968cb-6g8hl"]
Sep 10 12:28:38.947554 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:38.947522    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" path="/var/lib/kubelet/pods/c04bc52d-e225-4c52-ad2b-e1297f64be8f/volumes"
Sep 10 12:28:56.442601 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:56.442571    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-529mx_d2a3e6a9-9ae1-4180-aff2-a12e18b7411c/copy-manifests/0.log"
Sep 10 12:28:56.449975 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:56.449944    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-529mx_d2a3e6a9-9ae1-4180-aff2-a12e18b7411c/manager/0.log"
Sep 10 12:28:57.151572 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:57.151537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-645cf68f76-d58lt_ebb089bf-4dfe-4561-b718-9311743ea944/maas-api/0.log"
Sep 10 12:28:57.375139 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:57.375107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ljxxw_f5f04697-fb53-463e-9a01-d4876999dd2c/postgres/0.log"
Sep 10 12:28:58.574869 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:58.574835    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ddstq_8230adc6-bfca-47f4-a839-511a3173f664/kuadrant-console-plugin/0.log"
Sep 10 12:28:59.670789 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:59.670754    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-znq8x_b1a4ffee-5e55-4680-a6c3-76654dabf583/kube-auth-proxy/0.log"
Sep 10 12:28:59.915983 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:28:59.915956    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-47wtr_f851025f-f34e-4ef8-8ea6-93a31b66a242/payload-pre-processing/0.log"
Sep 10 12:29:00.162763 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:00.162732    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54478f5d74-kjsrv_0cb09d4e-faf7-49d9-81e6-1d7504d65425/router/0.log"
Sep 10 12:29:00.515657 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:00.515564    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-82m82_7aca7ba3-7f84-4850-a4af-d869a9fd9161/main/0.log"
Sep 10 12:29:00.626271 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:00.626241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-np2t9_c715f9c4-4d08-42f4-a186-9aa60c0bcdd3/main/0.log"
Sep 10 12:29:00.970824 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:00.970789    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9wv44_13ea1a33-13c8-4859-9266-827d9d06445f/main/0.log"
Sep 10 12:29:09.772965 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:09.772932    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j6wq6_edc233d2-8734-410f-9add-9b30778726f8/global-pull-secret-syncer/0.log"
Sep 10 12:29:09.956348 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:09.956294    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtfn5_02aaa613-10e8-4aca-9d8c-7b78a008d92a/konnectivity-agent/0.log"
Sep 10 12:29:10.048120 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:10.048040    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-58.ec2.internal_7aa9e6d39be639df8387e0cdc021c44c/haproxy/0.log"
Sep 10 12:29:15.002663 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:15.002632    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ddstq_8230adc6-bfca-47f4-a839-511a3173f664/kuadrant-console-plugin/0.log"
Sep 10 12:29:16.605251 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.605224    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/alertmanager/0.log"
Sep 10 12:29:16.634462 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.634431    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/config-reloader/0.log"
Sep 10 12:29:16.660144 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.660106    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/kube-rbac-proxy-web/0.log"
Sep 10 12:29:16.687419 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.687392    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/kube-rbac-proxy/0.log"
Sep 10 12:29:16.713273 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.713198    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/kube-rbac-proxy-metric/0.log"
Sep 10 12:29:16.745024 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.744998    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/prom-label-proxy/0.log"
Sep 10 12:29:16.769668 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.769645    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b7316fb9-7bd9-47f3-b4ac-c26f12c01f65/init-config-reloader/0.log"
Sep 10 12:29:16.860544 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.860506    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-8vjbb_64b45048-9461-44fa-b098-e6026ebd912d/kube-state-metrics/0.log"
Sep 10 12:29:16.884516 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.884486    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-8vjbb_64b45048-9461-44fa-b098-e6026ebd912d/kube-rbac-proxy-main/0.log"
Sep 10 12:29:16.913622 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:16.913597    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-8vjbb_64b45048-9461-44fa-b098-e6026ebd912d/kube-rbac-proxy-self/0.log"
Sep 10 12:29:17.361675 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.361649    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx694_d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad/node-exporter/0.log"
Sep 10 12:29:17.507767 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.507688    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx694_d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad/kube-rbac-proxy/0.log"
Sep 10 12:29:17.601706 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.601680    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx694_d68fea3f-b4c0-4e0f-8c0a-87c8774f49ad/init-textfile/0.log"
Sep 10 12:29:17.676757 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.676725    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gwvrx_48f4b857-a978-4980-bd89-5868b5b0dff8/kube-rbac-proxy-main/0.log"
Sep 10 12:29:17.709236 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.709211    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gwvrx_48f4b857-a978-4980-bd89-5868b5b0dff8/kube-rbac-proxy-self/0.log"
Sep 10 12:29:17.736329 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.736283    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-gwvrx_48f4b857-a978-4980-bd89-5868b5b0dff8/openshift-state-metrics/0.log"
Sep 10 12:29:17.790811 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.790725    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/prometheus/0.log"
Sep 10 12:29:17.819667 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.819635    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/config-reloader/0.log"
Sep 10 12:29:17.852731 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.852707    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/thanos-sidecar/0.log"
Sep 10 12:29:17.879537 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.879511    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/kube-rbac-proxy-web/0.log"
Sep 10 12:29:17.909825 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.909800    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/kube-rbac-proxy/0.log"
Sep 10 12:29:17.933851 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.933828    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/kube-rbac-proxy-thanos/0.log"
Sep 10 12:29:17.961183 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:17.961164    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1869aad1-aaaf-492e-a4ad-fb0bb9f4a06b/init-config-reloader/0.log"
Sep 10 12:29:18.206079 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.206054    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/thanos-query/0.log"
Sep 10 12:29:18.243224 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.243188    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/kube-rbac-proxy-web/0.log"
Sep 10 12:29:18.268619 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.268594    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/kube-rbac-proxy/0.log"
Sep 10 12:29:18.295800 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.295772    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/prom-label-proxy/0.log"
Sep 10 12:29:18.322040 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.321976    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/kube-rbac-proxy-rules/0.log"
Sep 10 12:29:18.362295 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:18.362268    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d9794884-c55p7_4af3e680-8418-4c18-9781-c249db2c0ff3/kube-rbac-proxy-metrics/0.log"
Sep 10 12:29:19.278094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.278059    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"]
Sep 10 12:29:19.278512 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.278471    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" containerName="maas-api"
Sep 10 12:29:19.278512 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.278484    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" containerName="maas-api"
Sep 10 12:29:19.278586 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.278579    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04bc52d-e225-4c52-ad2b-e1297f64be8f" containerName="maas-api"
Sep 10 12:29:19.281775 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.281755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.289780 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.289748    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xpjf\"/\"kube-root-ca.crt\""
Sep 10 12:29:19.290360 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.290343    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xpjf\"/\"openshift-service-ca.crt\""
Sep 10 12:29:19.290409 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.290364    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xpjf\"/\"default-dockercfg-k9vmw\""
Sep 10 12:29:19.302253 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.302214    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"]
Sep 10 12:29:19.381562 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.381529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-proc\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.381759 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.381582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-lib-modules\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.381759 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.381688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-sys\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.381759 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.381717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-podres\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.381759 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.381739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snnl\" (UniqueName: \"kubernetes.io/projected/8b3ece57-8294-4033-af11-547e749ec588-kube-api-access-9snnl\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482650 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482611    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-lib-modules\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-sys\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-podres\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snnl\" (UniqueName: \"kubernetes.io/projected/8b3ece57-8294-4033-af11-547e749ec588-kube-api-access-9snnl\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-proc\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-lib-modules\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.482841 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482815    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-sys\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.483094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-proc\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.483094 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.482879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b3ece57-8294-4033-af11-547e749ec588-podres\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.493129 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.493104    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snnl\" (UniqueName: \"kubernetes.io/projected/8b3ece57-8294-4033-af11-547e749ec588-kube-api-access-9snnl\") pod \"perf-node-gather-daemonset-2zrmm\" (UID: \"8b3ece57-8294-4033-af11-547e749ec588\") " pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.592648 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.592559    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:19.879335 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:19.879288    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"]
Sep 10 12:29:19.881277 ip-10-0-134-58 kubenswrapper[2573]: W0910 12:29:19.881253    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b3ece57_8294_4033_af11_547e749ec588.slice/crio-56dad577e3e0f6a5150a7c76d60fd5427ce5607109ae044ee4272fd6ebf08dfb WatchSource:0}: Error finding container 56dad577e3e0f6a5150a7c76d60fd5427ce5607109ae044ee4272fd6ebf08dfb: Status 404 returned error can't find the container with id 56dad577e3e0f6a5150a7c76d60fd5427ce5607109ae044ee4272fd6ebf08dfb
Sep 10 12:29:20.306428 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.306398    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/1.log"
Sep 10 12:29:20.310927 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.310908    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hwhk6_7bca6299-3fa1-4da5-9bc7-5fd01d13a4fc/console-operator/2.log"
Sep 10 12:29:20.349025 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.348994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm" event={"ID":"8b3ece57-8294-4033-af11-547e749ec588","Type":"ContainerStarted","Data":"d53fb9a23fd5fe83ae3c6110191d92690aa83186d8413801242673c7df97de44"}
Sep 10 12:29:20.349157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.349049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm" event={"ID":"8b3ece57-8294-4033-af11-547e749ec588","Type":"ContainerStarted","Data":"56dad577e3e0f6a5150a7c76d60fd5427ce5607109ae044ee4272fd6ebf08dfb"}
Sep 10 12:29:20.349157 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.349135    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:20.368691 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:20.368644    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm" podStartSLOduration=1.3686292930000001 podStartE2EDuration="1.368629293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:29:20.366816815 +0000 UTC m=+2261.946744230" watchObservedRunningTime="2026-09-10 12:29:20.368629293 +0000 UTC m=+2261.948556679"
Sep 10 12:29:22.743518 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:22.743484    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgs5d_a486eacf-3b31-4de8-9a5d-fcbff00af988/dns/0.log"
Sep 10 12:29:22.768457 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:22.768426    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cgs5d_a486eacf-3b31-4de8-9a5d-fcbff00af988/kube-rbac-proxy/0.log"
Sep 10 12:29:22.917284 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:22.917248    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zrfl6_f0632457-1698-4ba2-94de-33bf5e97f4af/dns-node-resolver/0.log"
Sep 10 12:29:23.522207 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:23.522166    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5xr7v_5bee52f2-0f1c-4c43-bb6b-0520c65a8af0/node-ca/0.log"
Sep 10 12:29:24.920959 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:24.920916    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-znq8x_b1a4ffee-5e55-4680-a6c3-76654dabf583/kube-auth-proxy/0.log"
Sep 10 12:29:25.011180 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:25.011151    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-47wtr_f851025f-f34e-4ef8-8ea6-93a31b66a242/payload-pre-processing/0.log"
Sep 10 12:29:25.081586 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:25.081552    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54478f5d74-kjsrv_0cb09d4e-faf7-49d9-81e6-1d7504d65425/router/0.log"
Sep 10 12:29:25.988469 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:25.988442    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hxmcd_cb874de1-6601-4984-9f35-f0a9c30e9d3d/serve-healthcheck-canary/0.log"
Sep 10 12:29:26.369807 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:26.369734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xpjf/perf-node-gather-daemonset-2zrmm"
Sep 10 12:29:26.551317 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:26.551268    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwhf7_8be26753-c6b2-4a1f-a7e9-9beb72381360/kube-rbac-proxy/0.log"
Sep 10 12:29:26.577421 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:26.577393    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwhf7_8be26753-c6b2-4a1f-a7e9-9beb72381360/exporter/0.log"
Sep 10 12:29:26.601471 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:26.601450    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwhf7_8be26753-c6b2-4a1f-a7e9-9beb72381360/extractor/0.log"
Sep 10 12:29:29.751499 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:29.751472    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-529mx_d2a3e6a9-9ae1-4180-aff2-a12e18b7411c/manager/0.log"
Sep 10 12:29:29.774825 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:29.774759    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d86dfbd7c-529mx_d2a3e6a9-9ae1-4180-aff2-a12e18b7411c/copy-manifests/0.log"
Sep 10 12:29:30.655533 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:30.655506    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-znq8x_b1a4ffee-5e55-4680-a6c3-76654dabf583/kube-auth-proxy/0.log"
Sep 10 12:29:30.720087 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:30.720056    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-47wtr_f851025f-f34e-4ef8-8ea6-93a31b66a242/payload-pre-processing/0.log"
Sep 10 12:29:30.764171 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:30.764144    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54478f5d74-kjsrv_0cb09d4e-faf7-49d9-81e6-1d7504d65425/router/0.log"
Sep 10 12:29:31.407250 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:31.407221    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f7f7d474c-7vr5h_83808e2b-aafd-4c4e-936f-94358b2fd12a/manager/0.log"
Sep 10 12:29:32.499016 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:32.498986    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5986dd8d8b-znq8x_b1a4ffee-5e55-4680-a6c3-76654dabf583/kube-auth-proxy/0.log"
Sep 10 12:29:32.561385 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:32.561360    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-47wtr_f851025f-f34e-4ef8-8ea6-93a31b66a242/payload-pre-processing/0.log"
Sep 10 12:29:32.607724 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:32.607695    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54478f5d74-kjsrv_0cb09d4e-faf7-49d9-81e6-1d7504d65425/router/0.log"
Sep 10 12:29:37.112088 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:37.112059    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hr2b2_a2c142af-6a36-4984-ba8f-8255afadd1e4/kube-storage-version-migrator-operator/1.log"
Sep 10 12:29:37.113122 ip-10-0-134-58 kubenswrapper[2573]: I0910 12:29:37.113087    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-hr2b2_a2c142af-6a36-4984-ba8f-8255afadd1e4/kube-storage-version-migrator-operator/0.log"