Sep 09 23:46:46.165175 ip-10-0-135-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 23:46:46.165185 ip-10-0-135-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 23:46:46.165192 ip-10-0-135-244 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 23:46:46.165418 ip-10-0-135-244 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 23:46:56.263296 ip-10-0-135-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 23:46:56.263317 ip-10-0-135-244 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 986cc5359bc8424aa428a191e1686c37 --
Sep 09 23:48:46.571858 ip-10-0-135-244 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 23:48:47.000501 ip-10-0-135-244 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:47.000501 ip-10-0-135-244 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 23:48:47.000501 ip-10-0-135-244 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:47.000501 ip-10-0-135-244 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 23:48:47.000501 ip-10-0-135-244 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:47.003479 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.003393    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 23:48:47.008497 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008482    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.008497 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008499    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008502    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008507    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008511    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008514    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008516    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008520    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008523    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008526    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008529    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008554    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008558    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008561    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008564    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008567    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008570    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008573    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008577    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008581    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.008576 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008584    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008588    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008591    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008594    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008596    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008599    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008602    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008605    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008609    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008612    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008615    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008627    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008630    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008633    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008636    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008639    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008642    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008645    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008647    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008649    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.009033 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008652    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008655    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008657    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008659    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008662    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008665    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008667    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008670    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008672    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008675    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008677    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008680    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008682    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008685    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008687    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008690    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008693    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008696    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008698    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008700    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.009529 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008703    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008706    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008708    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008711    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008714    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008717    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008720    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008723    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008725    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008728    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008730    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008733    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008735    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008738    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008740    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008743    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008745    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008748    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008750    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008752    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.010082 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008755    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008757    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008760    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008763    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008766    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008768    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008771    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008775    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008778    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008780    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008783    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008785    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008788    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008792    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008795    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008798    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008801    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008804    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.008807    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.010594 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009478    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009484    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009487    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009490    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009492    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009495    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009500    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009504    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009506    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009509    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009511    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009514    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009517    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009520    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009522    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009525    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009527    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009531    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009534    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.011047 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009556    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009563    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009568    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009572    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009575    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009578    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009581    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009583    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009586    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009589    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009592    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009594    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009599    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009601    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009604    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009606    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009609    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009611    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009614    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.011523 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009616    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009619    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009621    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009623    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009626    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009628    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009630    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009633    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009635    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009638    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009640    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009645    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009648    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009651    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009654    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009657    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009660    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009677    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009681    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009684    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.012000 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009686    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009689    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009692    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009695    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009698    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009701    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009704    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009707    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009709    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009712    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009714    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009717    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009720    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009722    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009725    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009727    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009729    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009732    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009734    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009737    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.012498 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009739    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009742    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009745    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009747    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009750    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009753    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009755    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009759    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009762    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009764    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009767    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009769    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009772    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009775    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009778    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009781    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009783    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009785    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009788    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009791    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.013110 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.009794    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010143    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010149    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010154    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010157    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010160    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010163    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010166    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010169    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010173    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010176    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010179    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010182    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010186    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010189    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010192    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010194    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010197    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.013610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010200    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010203    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010205    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010208    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010210    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010213    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010216    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010218    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010221    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010223    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010226    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010228    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010231    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010235    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010237    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010241    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010244    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010246    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010249    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010251    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.014040 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010254    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010256    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010259    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010261    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010264    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010266    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010268    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010271    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010273    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010276    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010278    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010281    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010284    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010286    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010289    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010292    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010294    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010296    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010299    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010302    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.014547 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010304    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010306    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010309    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010313    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010316    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010318    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010321    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010324    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010326    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010329    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010331    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010333    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010336    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010338    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010341    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010343    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010346    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010348    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010350    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010353    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.015068 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010356    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010359    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010361    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010364    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010367    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010369    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010372    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010374    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010376    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010379    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010382    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010384    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010387    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010389    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010392    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010395    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010398    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010400    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010403    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010408    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.015605 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010410    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.010413    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010489    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010531    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010556    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010561    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010566    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010569    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010573    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010578    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010581    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010583    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010587    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010590    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010593    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010597    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010599    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010603    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010605    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010608    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010613    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010615    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010619    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010622    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 23:48:47.016095 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010625    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010629    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010632    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010636    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010640    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010643    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010646    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010649    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010653    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010655    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010659    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010662    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010665    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010668    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010671    2576 flags.go:64] FLAG: --enable-server="true"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010674    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010678    2576 flags.go:64] FLAG: --event-burst="100"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010681    2576 flags.go:64] FLAG: --event-qps="50"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010685    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010688    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010691    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010695    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010698    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010701    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010704    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 09 23:48:47.016700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010706    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010709    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010712    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010715    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010717    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010720    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010723    2576 flags.go:64] FLAG: --feature-gates=""
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010727    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010730    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010733    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010738    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010741    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010743    2576 flags.go:64] FLAG: --help="false"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010746    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010749    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010752    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010755    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010758    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010762    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010765    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010767    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010770    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010773    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 23:48:47.017300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010775    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010779    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010783    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010785    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010788    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010791    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010794    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010797    2576 flags.go:64] FLAG: --lock-file=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010799    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010802    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010805    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010810    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010813    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010816    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010819    2576 flags.go:64] FLAG: --logging-format="text"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010822    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010825    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010828    2576 flags.go:64] FLAG: --manifest-url=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010830    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010835    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010838    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010842    2576 flags.go:64] FLAG: --max-pods="110"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010844    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010847    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010850    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 23:48:47.017904 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010853    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010856    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010859    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010862    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010869    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010872    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010875    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010878    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010881    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010886    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010890    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010893    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010896    2576 flags.go:64] FLAG: --port="10250"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010899    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010902    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00b048deacae36dda"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010904    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010907    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010910    2576 flags.go:64] FLAG: --register-node="true"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010913    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010916    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010919    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010922    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010925    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010928    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010931    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 23:48:47.018667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010934    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010938    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010941    2576 flags.go:64] FLAG: --runonce="false"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010944    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010947    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010950    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010952    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010955    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010958    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010961    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010964    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010967    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010970    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010973    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010976    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010978    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010981    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010985    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010990    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010993    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.010996    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011000    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011003    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011005    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011008    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 23:48:47.019770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011011    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011014    2576 flags.go:64] FLAG: --v="2"
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011018    2576 flags.go:64] FLAG: --version="false"
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011022    2576 flags.go:64] FLAG: --vmodule=""
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011025    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.011029    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011141    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011146    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011151    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011155    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011158    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011161    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011164    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011167    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011169    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011172    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011175    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011178    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011180    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011183    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011185    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.020801 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011188    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011190    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011193    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011198    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011202    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011205    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011208    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011210    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011213    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011216    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011218    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011221    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011224    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011226    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011229    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011231    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011234    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011237    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011240    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011244    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.021502 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011246    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011249    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011251    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011254    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011257    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011259    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011262    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011264    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011267    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011270    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011273    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011275    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011278    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011280    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011283    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011287    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011289    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011292    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011294    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011297    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.022007 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011299    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011302    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011304    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011307    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011309    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011312    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011314    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011317    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011319    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011322    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011324    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011329    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011332    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011334    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011336    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011339    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011342    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011344    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011347    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011349    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.022505 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011352    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011354    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011357    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011361    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011365    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011368    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011371    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011374    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011377    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011380    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011382    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011384    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011387    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011390    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011392    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011394    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011397    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011399    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011402    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.023098 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011404    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011407    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011409    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011412    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.011415    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.012173    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.021792    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.021806    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.022556    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.022563    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022636    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022642    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022645    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022648    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022651    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.023570 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022654    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022657    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022660    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022663    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022665    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022668    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022671    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022673    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022676    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022679    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022681    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022684    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022687    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022689    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022691    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022694    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022696    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022699    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022702    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022704    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.023948 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022707    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022714    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022717    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022719    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022723    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022725    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022728    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022731    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022733    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022736    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022738    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022741    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022744    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022746    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022749    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022752    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022754    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022756    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022759    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022762    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.024427 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022764    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022769    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022771    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022774    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022776    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022779    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022781    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022784    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022787    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022789    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022792    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022794    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022796    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022799    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022802    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022805    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022807    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022810    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022813    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.024925 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022815    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022818    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022820    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022823    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022827    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022830    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022834    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022837    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022840    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022843    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022846    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022848    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022851    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022854    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022857    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022860    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022862    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022864    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.025377 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022867    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022871    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022874    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022877    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022879    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022882    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022885    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022887    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022890    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022892    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022895    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022898    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022901    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022903    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022906    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022908    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.025839 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.022911    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.022916    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023031    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023037    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023041    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023044    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023047    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023050    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023053    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023056    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023059    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023062    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023064    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023067    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023070    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:47.026226 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023072    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023075    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023077    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023080    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023082    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023085    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023087    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023090    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023094    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023097    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023100    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023103    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023106    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023109    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023112    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023114    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023117    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023120    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023122    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:47.026610 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023125    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023127    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023130    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023133    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023135    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023138    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023141    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023143    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023145    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023148    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023150    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023153    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023155    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023158    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023161    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023163    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023166    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023168    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023171    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023173    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:47.027075 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023176    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023178    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023181    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023184    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023187    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023189    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023192    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023194    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023196    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023199    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023202    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023204    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023206    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023209    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023211    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023214    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023217    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023219    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023221    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:47.027563 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023224    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023226    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023229    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023231    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023234    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023237    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023240    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023243    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023245    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023248    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023250    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023253    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023255    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023257    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023260    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023262    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023265    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023268    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023270    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:47.028024 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023272    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023275    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023277    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023280    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023282    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023284    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023287    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023289    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:47.023292    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.023296    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.023403    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.026870    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.027975    2576 server.go:1019] "Starting client certificate rotation"
Sep 09 23:48:47.028532 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.028076    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 23:48:47.028885 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.028852    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 23:48:47.030776 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.030763    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 23:48:47.036243 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.036227    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 23:48:47.044815 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.044800    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 09 23:48:47.048383 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.048369    2576 log.go:25] "Validated CRI v1 image API"
Sep 09 23:48:47.050204 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.050187    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 23:48:47.054587 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.054569    2576 fs.go:135] Filesystem UUIDs: map[258c473e-2d8d-408d-895c-b33f5ad04526:/dev/nvme0n1p4 347a8ed1-b172-4199-8e66-f5e248ee90bb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 23:48:47.054635 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.054587    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 23:48:47.058452 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.058429    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 23:48:47.061798 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.061689    2576 manager.go:216] Machine: {Timestamp:2026-09-09 23:48:47.059744592 +0000 UTC m=+0.380120650 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097999 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22658396ef27cd01747bbdeb4c3142 SystemUUID:ec226583-96ef-27cd-0174-7bbdeb4c3142 BootID:986cc535-9bc8-424a-a428-a191e1686c37 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:c0:63:bd:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:c0:63:bd:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:38:52:5d:55:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 23:48:47.061798 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.061789    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 23:48:47.061912 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.061870    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 23:48:47.063113 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.063093    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 23:48:47.063242 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.063116    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 23:48:47.063963 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.063952    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 23:48:47.064004 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.063964    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 23:48:47.064004 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.063972    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 23:48:47.064674 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.064664    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 23:48:47.066588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.066579    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 23:48:47.071253 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071242    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 23:48:47.071365 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071355    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 23:48:47.071404 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071384    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 23:48:47.071404 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071393    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 23:48:47.071475 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071406    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 23:48:47.071475 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071416    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 09 23:48:47.071475 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071426    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 23:48:47.071778 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071765    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 23:48:47.071815 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.071794    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 23:48:47.074099 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.074073    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 23:48:47.076935 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.076921    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 23:48:47.076984 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.076951    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 23:48:47.077109 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077093    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 23:48:47.077109 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077111    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 23:48:47.077220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077117    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 23:48:47.077220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077122    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 23:48:47.077220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077130    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 23:48:47.077220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077135    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 23:48:47.077814 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077805    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 23:48:47.077814 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077814    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 23:48:47.077874 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077820    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 23:48:47.077874 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077826    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 23:48:47.077874 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077835    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 23:48:47.077874 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.077842    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 23:48:47.079299 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.079286    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 23:48:47.079299 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.079297    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 23:48:47.080832 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.080805    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 23:48:47.080909 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.080810    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 23:48:47.082151 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.082136    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jtfk"
Sep 09 23:48:47.083079 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.083068    2576 server.go:1268] "Started kubelet"
Sep 09 23:48:47.083187 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.083140    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 23:48:47.083187 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.083164    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 23:48:47.083287 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.083208    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 23:48:47.083926 ip-10-0-135-244 systemd[1]: Started Kubernetes Kubelet.
Sep 09 23:48:47.085279 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.085135    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 23:48:47.085339 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.085282    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 23:48:47.089856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.089840    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jtfk"
Sep 09 23:48:47.089933 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.089874    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 23:48:47.090163 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.090146    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 23:48:47.090237 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.090165    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 23:48:47.090237 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.090227    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.090932 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.090916    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 23:48:47.091069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091035    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 23:48:47.091069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091068    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 23:48:47.091225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091198    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 23:48:47.091225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091208    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 23:48:47.091458 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091405    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 23:48:47.091557 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091525    2576 factory.go:55] Registering systemd factory
Sep 09 23:48:47.091595 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091560    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 09 23:48:47.091782 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091769    2576 factory.go:153] Registering CRI-O factory
Sep 09 23:48:47.091821 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091785    2576 factory.go:223] Registration of the crio container factory successfully
Sep 09 23:48:47.091854 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091828    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 23:48:47.091854 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091847    2576 factory.go:103] Registering Raw factory
Sep 09 23:48:47.091913 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.091859    2576 manager.go:1196] Started watching for new ooms in manager
Sep 09 23:48:47.092296 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.092282    2576 manager.go:349] Starting recovery of all containers
Sep 09 23:48:47.092463 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.091305    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-244.ec2.internal.18d3cbf44ac410cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-244.ec2.internal,UID:ip-10-0-135-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-244.ec2.internal,},FirstTimestamp:2026-09-09 23:48:47.083040973 +0000 UTC m=+0.403417028,LastTimestamp:2026-09-09 23:48:47.083040973 +0000 UTC m=+0.403417028,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-244.ec2.internal,}"
Sep 09 23:48:47.097256 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.096977    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 23:48:47.097343 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.097275    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 23:48:47.105198 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.105183    2576 manager.go:354] Recovery completed
Sep 09 23:48:47.109511 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.109499    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.111465 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.111446    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.111557 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.111479    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.111557 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.111489    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.112015 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112004    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.112087 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112006    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 23:48:47.112087 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112025    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 23:48:47.112087 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112039    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 23:48:47.112744 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112729    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.112809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112759    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.112809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.112775    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.114124 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.114109    2576 policy_none.go:49] "None policy: Start"
Sep 09 23:48:47.114124 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.114126    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 23:48:47.114227 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.114136    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 23:48:47.116133 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.116123    2576 policy_none.go:47] "Start"
Sep 09 23:48:47.152152 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152130    2576 manager.go:337] "Starting Device Plugin manager"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.152175    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152189    2576 server.go:87] "Starting device plugin registration server"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152559    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152571    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152690    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152770    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.152781    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.153282    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 23:48:47.171503 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.153333    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.194040 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.194012    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 23:48:47.195345 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.195331    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 23:48:47.195421 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.195373    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 23:48:47.195421 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.195393    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 23:48:47.195522 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.195433    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 23:48:47.197636 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.197615    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:47.253631 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.253571    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.254364 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.254351    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.254416 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.254381    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.254416 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.254395    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.254416 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.254416    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.259949 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.259935    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.260004 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.259955    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-244.ec2.internal\": node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.276510 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.276487    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.296278 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.296252    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"]
Sep 09 23:48:47.296354 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.296316    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.297196 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.297180    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.297261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.297212    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.297261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.297225    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.299534 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.299520    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.299652 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.299637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.299688 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.299671    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.300268 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300242    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.300268 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300269    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.300396 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300281    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.300396 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300332    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.300396 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300352    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.300396 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.300384    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.302526 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.302510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.302632 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.302554    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:47.304181 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.304162    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:47.304264 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.304188    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:47.304264 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.304202    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:47.327103 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.327086    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-244.ec2.internal\" not found" node="ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.330529 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.330515    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-244.ec2.internal\" not found" node="ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.377372 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.377352    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.393970 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.393947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.394054 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.393976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.394054 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.394000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5b08fa5b23f3cccc007bda111536ab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"bd5b08fa5b23f3cccc007bda111536ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.477619 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.477590    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.494993 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.494972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.495062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.495000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5b08fa5b23f3cccc007bda111536ab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"bd5b08fa5b23f3cccc007bda111536ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.495062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.495017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.495062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.495044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.495183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.495081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c542b7a0479fd18f5b82c43a08a02a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"3c542b7a0479fd18f5b82c43a08a02a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.495183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.495083    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd5b08fa5b23f3cccc007bda111536ab-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"bd5b08fa5b23f3cccc007bda111536ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.578371 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.578347    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.629913 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.629887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.633046 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:47.633027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:47.678853 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.678825    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.779362 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.779333    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.879929 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.879873    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:47.980409 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:47.980388    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:48.027893 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.027867    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 23:48:48.028286 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.028010    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 23:48:48.080893 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:48.080872    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:48.091125 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.091108    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:48.096995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.096969    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:43:47 +0000 UTC" deadline="2028-04-18 20:00:21.850338864 +0000 UTC"
Sep 09 23:48:48.097073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.096996    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14084h11m33.753347855s"
Sep 09 23:48:48.181100 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:48.181067    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:48.207784 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.207764    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:48.281694 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:48.281668    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 09 23:48:48.299646 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:48.299621    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5b08fa5b23f3cccc007bda111536ab.slice/crio-6789e674f0d7923630db392c86d60e7a704d0d466333efc5d38832e8bd8caaf1 WatchSource:0}: Error finding container 6789e674f0d7923630db392c86d60e7a704d0d466333efc5d38832e8bd8caaf1: Status 404 returned error can't find the container with id 6789e674f0d7923630db392c86d60e7a704d0d466333efc5d38832e8bd8caaf1
Sep 09 23:48:48.299977 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:48.299953    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c542b7a0479fd18f5b82c43a08a02a7.slice/crio-d9cefaabf91102548073c310c874dc8687a3f468ad3317aa452e8987709210fb WatchSource:0}: Error finding container d9cefaabf91102548073c310c874dc8687a3f468ad3317aa452e8987709210fb: Status 404 returned error can't find the container with id d9cefaabf91102548073c310c874dc8687a3f468ad3317aa452e8987709210fb
Sep 09 23:48:48.305351 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.305337    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:48:48.315780 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.315756    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:48.390534 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.390514    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:48.402213 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.402171    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 23:48:48.402965 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.402954    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 09 23:48:48.412376 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:48.412361    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 23:48:49.072858 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.072603    2576 apiserver.go:52] "Watching apiserver"
Sep 09 23:48:49.081009 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.080981    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 23:48:49.082188 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.082164    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal","openshift-multus/multus-z5qmv","openshift-multus/network-metrics-daemon-ttqwg","kube-system/global-pull-secret-syncer-2prt6","openshift-dns/node-resolver-hfzpf","openshift-multus/multus-additional-cni-plugins-bwtfb","openshift-network-diagnostics/network-check-target-m7j6c","openshift-network-operator/iptables-alerter-lj8x5","openshift-ovn-kubernetes/ovnkube-node-st67z","kube-system/konnectivity-agent-pr48t","kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c","openshift-cluster-node-tuning-operator/tuned-pmvmj","openshift-image-registry/node-ca-n8f6b"]
Sep 09 23:48:49.085233 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.085161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:49.085361 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.085248    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:49.087551 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.087288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.089863 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.089843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87jk9\""
Sep 09 23:48:49.090082 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.090067    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.090265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.090251    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.090812 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.090710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.090924 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.090887    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:49.090924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.090718    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 23:48:49.092205 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.092184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.092309 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.092258    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:48:49.094830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.094813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.097709 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.097655    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.097709 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.097687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.097986 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.097966    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p7ksc\""
Sep 09 23:48:49.098234 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.098185    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.100158 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.100126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.100657 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.100636    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.100835 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.100821    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 23:48:49.101002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.100985    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 23:48:49.101110 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.100988    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 23:48:49.101263 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.101246    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.101381 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.101364    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-276v9\""
Sep 09 23:48:49.102667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.102645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.103651 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.103614    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 23:48:49.104809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.104784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 23:48:49.105755 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysconfig\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.105855 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-kubernetes\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.105855 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-tuned\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.105855 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-dbus\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.106002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:49.106002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-sys\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-lib-modules\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-var-lib-kubelet\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106002 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.105996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-host\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-kubelet-config\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-conf\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-systemd\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-run\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-tmp\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55sn\" (UniqueName: \"kubernetes.io/projected/28c0d142-d65a-4243-9dbc-088ec2d506e1-kube-api-access-s55sn\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106560 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.106560 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-modprobe-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106560 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.106560 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.106359    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cs76\" (UniqueName: \"kubernetes.io/projected/7698041c-72bd-4509-840e-9add393b3bff-kube-api-access-8cs76\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.107302 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.107282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.108968 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.108941    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.109068 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109055    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 23:48:49.109130 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109111    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 23:48:49.109193 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.108946    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.109427 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109386    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 23:48:49.109604 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109587    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 23:48:49.109688 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109612    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nm8xx\""
Sep 09 23:48:49.109688 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.109682    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.110043 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.110020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.110240 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.110218    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.110426 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.110410    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxd9l\""
Sep 09 23:48:49.111439 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.111415    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 23:48:49.111568 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.111531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kxbph\""
Sep 09 23:48:49.112308 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.112292    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 23:48:49.113050 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.113027    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 23:48:49.113209 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.113032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8lr4\""
Sep 09 23:48:49.119148 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.119126    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.121757 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.121738    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.121856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.121765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.123155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.121942    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.123155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.122479    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kjppv\""
Sep 09 23:48:49.123155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.122599    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 23:48:49.123155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.122782    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkksv"
Sep 09 23:48:49.124236 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.124207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7drwl\""
Sep 09 23:48:49.124315 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.124302    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 23:48:49.124390 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.124370    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 23:48:49.124524 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.124500    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 23:48:49.131116 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.131062    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkksv"
Sep 09 23:48:49.192304 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.192281    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 23:48:49.199910 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.199864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"3c542b7a0479fd18f5b82c43a08a02a7","Type":"ContainerStarted","Data":"d9cefaabf91102548073c310c874dc8687a3f468ad3317aa452e8987709210fb"}
Sep 09 23:48:49.201109 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.201089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" event={"ID":"bd5b08fa5b23f3cccc007bda111536ab","Type":"ContainerStarted","Data":"6789e674f0d7923630db392c86d60e7a704d0d466333efc5d38832e8bd8caaf1"}
Sep 09 23:48:49.206823 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-kubelet\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.206924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/608f8332-85df-464d-bc29-23ed75a98c0a-host-slash\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.206924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f909248-ff01-4fa8-a721-245433a608ef-agent-certs\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.206924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cnibin\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.207028 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-conf\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.207028 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.206972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-sys-fs\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.207028 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-cni-binary-copy\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.207143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-k8s-cni-cncf-io\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.207143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-bin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.207143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-conf\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.207143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.207300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-netns\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.207441 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-log-socket\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.207565 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-tuned\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.207565 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d90c34-7f98-4289-99f4-32f0ecc6b581-tmp-dir\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.207565 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.207722 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.207584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-systemd-units\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.208323 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.208287    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.208922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-ovn\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.208978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-netns\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-sys\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrnp\" (UniqueName: \"kubernetes.io/projected/6bbacd63-8d59-408f-9ca0-b9b3b8993815-kube-api-access-ktrnp\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvgq\" (UniqueName: \"kubernetes.io/projected/608f8332-85df-464d-bc29-23ed75a98c0a-kube-api-access-kwvgq\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55sn\" (UniqueName: \"kubernetes.io/projected/28c0d142-d65a-4243-9dbc-088ec2d506e1-kube-api-access-s55sn\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-modprobe-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysctl-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-cnibin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-daemon-config\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhp7c\" (UniqueName: \"kubernetes.io/projected/54d90c34-7f98-4289-99f4-32f0ecc6b581-kube-api-access-rhp7c\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.211104 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ede7ec3-5176-498b-890d-bc4614e18301-serviceca\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-multus-certs\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-node-log\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-lib-modules\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ede7ec3-5176-498b-890d-bc4614e18301-host\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.209933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-env-overrides\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-etc-kubernetes\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.210095    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-lib-modules\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d90c34-7f98-4289-99f4-32f0ecc6b581-hosts-file\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cs76\" (UniqueName: \"kubernetes.io/projected/7698041c-72bd-4509-840e-9add393b3bff-kube-api-access-8cs76\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.210251    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:48:49.710205047 +0000 UTC m=+3.030581114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210256    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-modprobe-d\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-kubelet-config\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-sys\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.212183 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210671    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-systemd\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-run\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.210726    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-tmp\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-kubelet-config\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.210800    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:49.710777061 +0000 UTC m=+3.031153106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-systemd\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-hostroot\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxszq\" (UniqueName: \"kubernetes.io/projected/3ede7ec3-5176-498b-890d-bc4614e18301-kube-api-access-zxszq\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f909248-ff01-4fa8-a721-245433a608ef-konnectivity-ca\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-run\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.210943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-system-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-socket-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnxt\" (UniqueName: \"kubernetes.io/projected/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kube-api-access-mfnxt\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-slash\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213126 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-var-lib-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-bin\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-registration-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-dbus\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-var-lib-kubelet\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-host\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-host\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fc1b81b3-022d-4897-bf77-2c0b3b387446-dbus\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-netd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-var-lib-kubelet\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-conf-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbgq\" (UniqueName: \"kubernetes.io/projected/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-kube-api-access-tvbgq\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-socket-dir-parent\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.211946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpbp\" (UniqueName: \"kubernetes.io/projected/63c61f47-fa07-4006-abdb-e98e855f96a0-kube-api-access-sfpbp\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-system-cni-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-multus\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.213995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/608f8332-85df-464d-bc29-23ed75a98c0a-iptables-alerter-script\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysconfig\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-kubernetes\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-kubelet\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-systemd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-etc-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovn-node-metrics-cert\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-device-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-config\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-kubernetes\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-script-lib\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-sysconfig\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.212799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.214328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-etc-tuned\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.214692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.214372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28c0d142-d65a-4243-9dbc-088ec2d506e1-tmp\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.216640 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.216608    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:49.216640 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.216641    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:49.216764 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.216654    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:49.216764 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.216699    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:49.716687515 +0000 UTC m=+3.037063558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:49.218647 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.218622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55sn\" (UniqueName: \"kubernetes.io/projected/28c0d142-d65a-4243-9dbc-088ec2d506e1-kube-api-access-s55sn\") pod \"tuned-pmvmj\" (UID: \"28c0d142-d65a-4243-9dbc-088ec2d506e1\") " pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.218973 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.218951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cs76\" (UniqueName: \"kubernetes.io/projected/7698041c-72bd-4509-840e-9add393b3bff-kube-api-access-8cs76\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.313602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-node-log\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.313602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ede7ec3-5176-498b-890d-bc4614e18301-host\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-env-overrides\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-node-log\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-etc-kubernetes\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ede7ec3-5176-498b-890d-bc4614e18301-host\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d90c34-7f98-4289-99f4-32f0ecc6b581-hosts-file\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54d90c34-7f98-4289-99f4-32f0ecc6b581-hosts-file\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.313846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-etc-kubernetes\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-hostroot\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxszq\" (UniqueName: \"kubernetes.io/projected/3ede7ec3-5176-498b-890d-bc4614e18301-kube-api-access-zxszq\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f909248-ff01-4fa8-a721-245433a608ef-konnectivity-ca\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-system-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-socket-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.313932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-hostroot\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnxt\" (UniqueName: \"kubernetes.io/projected/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kube-api-access-mfnxt\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-slash\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-var-lib-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-bin\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-registration-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-netd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-conf-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.314261 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbgq\" (UniqueName: \"kubernetes.io/projected/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-kube-api-access-tvbgq\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-env-overrides\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-socket-dir-parent\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpbp\" (UniqueName: \"kubernetes.io/projected/63c61f47-fa07-4006-abdb-e98e855f96a0-kube-api-access-sfpbp\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-system-cni-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-multus\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/608f8332-85df-464d-bc29-23ed75a98c0a-iptables-alerter-script\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-system-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-kubelet\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-systemd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-conf-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-etc-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-socket-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3f909248-ff01-4fa8-a721-245433a608ef-konnectivity-ca\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314521    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-slash\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.315220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovn-node-metrics-cert\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-socket-dir-parent\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314650    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-var-lib-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-device-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-bin\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-system-cni-dir\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-cni-netd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-registration-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-device-dir\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314868    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-multus\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-config\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-script-lib\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-systemd\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-kubelet\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-etc-openvswitch\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.316337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.314995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-kubelet\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/608f8332-85df-464d-bc29-23ed75a98c0a-host-slash\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f909248-ff01-4fa8-a721-245433a608ef-agent-certs\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cnibin\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-sys-fs\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315152    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-kubelet\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/608f8332-85df-464d-bc29-23ed75a98c0a-host-slash\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315460    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-config\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315540    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-cni-dir\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315587    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cnibin\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-cni-binary-copy\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-k8s-cni-cncf-io\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-bin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/608f8332-85df-464d-bc29-23ed75a98c0a-iptables-alerter-script\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-netns\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.317269 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-log-socket\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-cni-binary-copy\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315788    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d90c34-7f98-4289-99f4-32f0ecc6b581-tmp-dir\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-sys-fs\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-systemd-units\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-ovn\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-netns\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315880    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-systemd-units\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrnp\" (UniqueName: \"kubernetes.io/projected/6bbacd63-8d59-408f-9ca0-b9b3b8993815-kube-api-access-ktrnp\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvgq\" (UniqueName: \"kubernetes.io/projected/608f8332-85df-464d-bc29-23ed75a98c0a-kube-api-access-kwvgq\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-cnibin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-daemon-config\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-var-lib-cni-bin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318069 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhp7c\" (UniqueName: \"kubernetes.io/projected/54d90c34-7f98-4289-99f4-32f0ecc6b581-kube-api-access-rhp7c\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-log-socket\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ede7ec3-5176-498b-890d-bc4614e18301-serviceca\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-multus-certs\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-run-ovn\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.315944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-netns\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-multus-certs\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316258    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d90c34-7f98-4289-99f4-32f0ecc6b581-tmp-dir\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-k8s-cni-cncf-io\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-cni-binary-copy\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316573    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-cnibin\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63c61f47-fa07-4006-abdb-e98e855f96a0-host-run-netns\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316755    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6bbacd63-8d59-408f-9ca0-b9b3b8993815-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.316956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63c61f47-fa07-4006-abdb-e98e855f96a0-multus-daemon-config\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.317001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovnkube-script-lib\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.318719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.317097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ede7ec3-5176-498b-890d-bc4614e18301-serviceca\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.319317 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.317841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-ovn-node-metrics-cert\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.319317 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.318378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3f909248-ff01-4fa8-a721-245433a608ef-agent-certs\") pod \"konnectivity-agent-pr48t\" (UID: \"3f909248-ff01-4fa8-a721-245433a608ef\") " pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.322492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.322459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnxt\" (UniqueName: \"kubernetes.io/projected/75aeaac5-084e-4991-bb7f-3c60d9f3bbaa-kube-api-access-mfnxt\") pod \"aws-ebs-csi-driver-node-4xt9c\" (UID: \"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.323450 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.323390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxszq\" (UniqueName: \"kubernetes.io/projected/3ede7ec3-5176-498b-890d-bc4614e18301-kube-api-access-zxszq\") pod \"node-ca-n8f6b\" (UID: \"3ede7ec3-5176-498b-890d-bc4614e18301\") " pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.323450 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.323393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpbp\" (UniqueName: \"kubernetes.io/projected/63c61f47-fa07-4006-abdb-e98e855f96a0-kube-api-access-sfpbp\") pod \"multus-z5qmv\" (UID: \"63c61f47-fa07-4006-abdb-e98e855f96a0\") " pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.323695 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.323525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbgq\" (UniqueName: \"kubernetes.io/projected/402bf08e-c18b-4a14-ace4-6c259ab7aa8c-kube-api-access-tvbgq\") pod \"ovnkube-node-st67z\" (UID: \"402bf08e-c18b-4a14-ace4-6c259ab7aa8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.324665 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.324644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhp7c\" (UniqueName: \"kubernetes.io/projected/54d90c34-7f98-4289-99f4-32f0ecc6b581-kube-api-access-rhp7c\") pod \"node-resolver-hfzpf\" (UID: \"54d90c34-7f98-4289-99f4-32f0ecc6b581\") " pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.325231 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.325210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrnp\" (UniqueName: \"kubernetes.io/projected/6bbacd63-8d59-408f-9ca0-b9b3b8993815-kube-api-access-ktrnp\") pod \"multus-additional-cni-plugins-bwtfb\" (UID: \"6bbacd63-8d59-408f-9ca0-b9b3b8993815\") " pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.325557 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.325532    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvgq\" (UniqueName: \"kubernetes.io/projected/608f8332-85df-464d-bc29-23ed75a98c0a-kube-api-access-kwvgq\") pod \"iptables-alerter-lj8x5\" (UID: \"608f8332-85df-464d-bc29-23ed75a98c0a\") " pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.399821 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.399779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pmvmj"
Sep 09 23:48:49.413713 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.413692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hfzpf"
Sep 09 23:48:49.428554 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.428512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bwtfb"
Sep 09 23:48:49.435618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.435593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:48:49.443458 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.443441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n8f6b"
Sep 09 23:48:49.451342 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.451323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5qmv"
Sep 09 23:48:49.458149 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.458132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:48:49.466160 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.466140    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:49.467203 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.467181    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lj8x5"
Sep 09 23:48:49.474158 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.474135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c"
Sep 09 23:48:49.719884 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.719795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:49.719884 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.719854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:49.719884 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:49.719882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.719982    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.719982    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720011    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720033    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:50.720017493 +0000 UTC m=+4.040393540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720035    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720047    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:48:50.720040806 +0000 UTC m=+4.040416850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720051    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:49.720170 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:49.720101    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:50.720082554 +0000 UTC m=+4.040458615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:50.134125 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.134078    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:49 +0000 UTC" deadline="2028-03-19 18:34:15.263800625 +0000 UTC"
Sep 09 23:48:50.134125 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.134108    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13362h45m25.129696814s"
Sep 09 23:48:50.195773 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.195733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:50.195929 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.195854    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:50.318211 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.318177    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75aeaac5_084e_4991_bb7f_3c60d9f3bbaa.slice/crio-f8a531abf0dcf026c510db3513877705dd366c4783db3b2f7d9dcb814b80bd92 WatchSource:0}: Error finding container f8a531abf0dcf026c510db3513877705dd366c4783db3b2f7d9dcb814b80bd92: Status 404 returned error can't find the container with id f8a531abf0dcf026c510db3513877705dd366c4783db3b2f7d9dcb814b80bd92
Sep 09 23:48:50.320632 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.320603    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c61f47_fa07_4006_abdb_e98e855f96a0.slice/crio-5b0e2bb8d08f5a663bed42e051f40dff27c5fa6cfca4f141312fc602e1e6603a WatchSource:0}: Error finding container 5b0e2bb8d08f5a663bed42e051f40dff27c5fa6cfca4f141312fc602e1e6603a: Status 404 returned error can't find the container with id 5b0e2bb8d08f5a663bed42e051f40dff27c5fa6cfca4f141312fc602e1e6603a
Sep 09 23:48:50.321890 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.321829    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f909248_ff01_4fa8_a721_245433a608ef.slice/crio-a4067f25860e9cca23d850b9ae9bf58b694fb8a65e74a73f45e6d8b47f4bc732 WatchSource:0}: Error finding container a4067f25860e9cca23d850b9ae9bf58b694fb8a65e74a73f45e6d8b47f4bc732: Status 404 returned error can't find the container with id a4067f25860e9cca23d850b9ae9bf58b694fb8a65e74a73f45e6d8b47f4bc732
Sep 09 23:48:50.322485 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.322446    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608f8332_85df_464d_bc29_23ed75a98c0a.slice/crio-e668e2145de46ec8bd39636f3fb4246475b71e299cb02d917f7bf89ae275ee23 WatchSource:0}: Error finding container e668e2145de46ec8bd39636f3fb4246475b71e299cb02d917f7bf89ae275ee23: Status 404 returned error can't find the container with id e668e2145de46ec8bd39636f3fb4246475b71e299cb02d917f7bf89ae275ee23
Sep 09 23:48:50.347687 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.347667    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ede7ec3_5176_498b_890d_bc4614e18301.slice/crio-d5a631af7933fd6b8859f881925cb0508ed362686230d3323651186d4605a4a4 WatchSource:0}: Error finding container d5a631af7933fd6b8859f881925cb0508ed362686230d3323651186d4605a4a4: Status 404 returned error can't find the container with id d5a631af7933fd6b8859f881925cb0508ed362686230d3323651186d4605a4a4
Sep 09 23:48:50.348349 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.348329    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d90c34_7f98_4289_99f4_32f0ecc6b581.slice/crio-096e68ce392c97da88f68ed711a775718eadc9a054dc51f492003eea5fdbc150 WatchSource:0}: Error finding container 096e68ce392c97da88f68ed711a775718eadc9a054dc51f492003eea5fdbc150: Status 404 returned error can't find the container with id 096e68ce392c97da88f68ed711a775718eadc9a054dc51f492003eea5fdbc150
Sep 09 23:48:50.349454 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.349434    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402bf08e_c18b_4a14_ace4_6c259ab7aa8c.slice/crio-2acb4119bc1ac5ff74a4b1b401367645c7ae4266f6e532527081fb663f9b72d9 WatchSource:0}: Error finding container 2acb4119bc1ac5ff74a4b1b401367645c7ae4266f6e532527081fb663f9b72d9: Status 404 returned error can't find the container with id 2acb4119bc1ac5ff74a4b1b401367645c7ae4266f6e532527081fb663f9b72d9
Sep 09 23:48:50.350366 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:48:50.350336    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c0d142_d65a_4243_9dbc_088ec2d506e1.slice/crio-d769f26ade198ac5a973a1b2bb3ab54208d51f7fa495cdecdf07beb9c50aaf2a WatchSource:0}: Error finding container d769f26ade198ac5a973a1b2bb3ab54208d51f7fa495cdecdf07beb9c50aaf2a: Status 404 returned error can't find the container with id d769f26ade198ac5a973a1b2bb3ab54208d51f7fa495cdecdf07beb9c50aaf2a
Sep 09 23:48:50.727195 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.727158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:50.727340 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.727209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:50.727340 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:50.727254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727346    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727351    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727377    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727380    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727390    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727412    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:48:52.727390636 +0000 UTC m=+6.047766698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:50.727447 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727439    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:52.727421617 +0000 UTC m=+6.047797680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:50.727872 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:50.727455    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:52.72744894 +0000 UTC m=+6.047824985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:51.135387 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.135326    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:49 +0000 UTC" deadline="2028-04-30 15:16:37.534537526 +0000 UTC"
Sep 09 23:48:51.135387 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.135381    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14367h27m46.399160151s"
Sep 09 23:48:51.197431 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.196747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:51.197431 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:51.196895    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:51.197910 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.197763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:51.197910 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:51.197871    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:48:51.219905 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.219872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerStarted","Data":"727c1acdb7a4a9663a63eb2c6187924b8c0e5b54e648ead931179152c4fc3354"}
Sep 09 23:48:51.235294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.235262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmvmj" event={"ID":"28c0d142-d65a-4243-9dbc-088ec2d506e1","Type":"ContainerStarted","Data":"d769f26ade198ac5a973a1b2bb3ab54208d51f7fa495cdecdf07beb9c50aaf2a"}
Sep 09 23:48:51.264586 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.264552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"2acb4119bc1ac5ff74a4b1b401367645c7ae4266f6e532527081fb663f9b72d9"}
Sep 09 23:48:51.276492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.276459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hfzpf" event={"ID":"54d90c34-7f98-4289-99f4-32f0ecc6b581","Type":"ContainerStarted","Data":"096e68ce392c97da88f68ed711a775718eadc9a054dc51f492003eea5fdbc150"}
Sep 09 23:48:51.281260 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.281230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lj8x5" event={"ID":"608f8332-85df-464d-bc29-23ed75a98c0a","Type":"ContainerStarted","Data":"e668e2145de46ec8bd39636f3fb4246475b71e299cb02d917f7bf89ae275ee23"}
Sep 09 23:48:51.292536 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.292508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n8f6b" event={"ID":"3ede7ec3-5176-498b-890d-bc4614e18301","Type":"ContainerStarted","Data":"d5a631af7933fd6b8859f881925cb0508ed362686230d3323651186d4605a4a4"}
Sep 09 23:48:51.301145 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.301097    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr48t" event={"ID":"3f909248-ff01-4fa8-a721-245433a608ef","Type":"ContainerStarted","Data":"a4067f25860e9cca23d850b9ae9bf58b694fb8a65e74a73f45e6d8b47f4bc732"}
Sep 09 23:48:51.316272 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.316192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5qmv" event={"ID":"63c61f47-fa07-4006-abdb-e98e855f96a0","Type":"ContainerStarted","Data":"5b0e2bb8d08f5a663bed42e051f40dff27c5fa6cfca4f141312fc602e1e6603a"}
Sep 09 23:48:51.329293 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.329263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c" event={"ID":"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa","Type":"ContainerStarted","Data":"f8a531abf0dcf026c510db3513877705dd366c4783db3b2f7d9dcb814b80bd92"}
Sep 09 23:48:51.342371 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.342344    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c542b7a0479fd18f5b82c43a08a02a7" containerID="55b19d4d0e8089fc52a64e841f9d5cc9d91a02dbba1564b1b95608f6e493f109" exitCode=0
Sep 09 23:48:51.342469 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.342439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"3c542b7a0479fd18f5b82c43a08a02a7","Type":"ContainerDied","Data":"55b19d4d0e8089fc52a64e841f9d5cc9d91a02dbba1564b1b95608f6e493f109"}
Sep 09 23:48:51.359945 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.359869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" event={"ID":"bd5b08fa5b23f3cccc007bda111536ab","Type":"ContainerStarted","Data":"492abe511d63772b3e62dd24877393403c2bbcd7696ac670d63f5345950995c7"}
Sep 09 23:48:51.376646 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:51.376102    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" podStartSLOduration=3.376083911 podStartE2EDuration="3.376083911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:51.375250951 +0000 UTC m=+4.695627023" watchObservedRunningTime="2026-09-09 23:48:51.376083911 +0000 UTC m=+4.696459980"
Sep 09 23:48:52.197188 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:52.196449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:52.197188 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.196811    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:52.377868 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:52.377831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"3c542b7a0479fd18f5b82c43a08a02a7","Type":"ContainerStarted","Data":"70950001d82f70adb37382d67bedb9c960c9a5c0980a86a860aa280b202deff6"}
Sep 09 23:48:52.747447 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:52.747409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:52.747466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:52.747528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747611    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747653    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:52.747709 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747656    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:52.748037 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747720    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:48:56.747698337 +0000 UTC m=+10.068074396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:52.748037 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747741    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:56.747729659 +0000 UTC m=+10.068105708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:52.748037 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747773    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:52.748037 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:52.747819    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:56.747800811 +0000 UTC m=+10.068176871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:53.196711 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:53.196676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:53.196914 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:53.196825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:53.196990 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:53.196941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:53.197110 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:53.197039    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:48:54.195651 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:54.195609    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:54.196102 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:54.195761    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:55.196630 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:55.196536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:55.197126 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:55.196697    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:48:55.197238 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:55.197196    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:55.197352 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:55.197330    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:56.195635 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:56.195605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:56.195821 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.195739    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:56.782226 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:56.782178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:56.782860 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:56.782235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:56.782860 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:56.782267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:56.782860 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.782382    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:56.782860 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.782441    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:04.782422853 +0000 UTC m=+18.102798913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.782864    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.782905    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:49:04.782893114 +0000 UTC m=+18.103269169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.782983    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.783000    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.783011    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:56.783056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:56.783037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:49:04.783029428 +0000 UTC m=+18.103405483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:57.198504 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:57.198468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:57.198689 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:57.198612    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:57.198971 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:57.198954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:57.199063 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:57.199045    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:48:58.195619 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:58.195585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:48:58.196056 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:58.195724    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:48:59.195983 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:59.195904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:48:59.196434 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:59.196012    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:48:59.196434 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:48:59.196294    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:48:59.196434 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:48:59.196371    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:00.196234 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:00.196206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:00.196623 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:00.196308    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:01.195714 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:01.195676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:01.195909 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:01.195676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:01.195909 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:01.195789    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:01.195909 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:01.195869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:02.196458 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:02.196418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:02.196915 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:02.196558    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:03.196110 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:03.196078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:03.196296 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:03.196218    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:03.196296 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:03.196276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:03.196412 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:03.196367    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:04.195600 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:04.195565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:04.196021 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.195684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:04.845793 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:04.845753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:04.845793 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:04.845801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845870    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845896    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845914    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845927    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845934    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:49:20.845916352 +0000 UTC m=+34.166292408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:04.845963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:04.846010 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.845974    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:49:20.845959704 +0000 UTC m=+34.166335767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:04.846307 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.846023    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:04.846307 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:04.846065    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:20.846054178 +0000 UTC m=+34.166430220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:05.196455 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:05.196374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:05.196455 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:05.196416    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:05.196960 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:05.196513    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:05.196960 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:05.196605    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:06.196196 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:06.196163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:06.196364 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:06.196274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:07.196386 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:07.196352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:07.196854 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:07.196453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:07.196854 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:07.196812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:07.196963 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:07.196905    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:08.196342 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:08.196307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:08.196516 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:08.196431    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:09.196394 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.196265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:09.196570 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:09.196489    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:09.197150 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.196288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:09.197150 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:09.197106    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:09.408661 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.408627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c" event={"ID":"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa","Type":"ContainerStarted","Data":"b42b84372c83854aa177c8c05b52226de1d6487cb6ac9c74b20c9d9a49b8db9d"}
Sep 09 23:49:09.410381 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.410348    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="fbbbb4acebaa661453412aa6a312f1dbf8c7a740b2c0e5d706a74de3e3ba7c04" exitCode=0
Sep 09 23:49:09.410478 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.410430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"fbbbb4acebaa661453412aa6a312f1dbf8c7a740b2c0e5d706a74de3e3ba7c04"}
Sep 09 23:49:09.412442 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.411991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pmvmj" event={"ID":"28c0d142-d65a-4243-9dbc-088ec2d506e1","Type":"ContainerStarted","Data":"70c35c6c33309a54f3dcc6ceff844452f274146d4bd622ff4a4cb94a4fbe664d"}
Sep 09 23:49:09.415588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.415559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"30df57eaaa6ca6371c458189686a7db82874eaf07da55e49c8969b97b60f95b9"}
Sep 09 23:49:09.415683 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.415644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"8619aa3daeec69fc00455487d8424f4abc06cbdf7829fbe1555254c5199d4a93"}
Sep 09 23:49:09.415730 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.415696    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"e28e7fecd1b8f16e1f01da4ea02b8857da4ee99653da00dd2268c0716e0338f6"}
Sep 09 23:49:09.417207 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.417175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hfzpf" event={"ID":"54d90c34-7f98-4289-99f4-32f0ecc6b581","Type":"ContainerStarted","Data":"ba36d3010e467189e9b28888c9bd9cd47ec826422adf0c00c009e42393e1b704"}
Sep 09 23:49:09.418874 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.418848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n8f6b" event={"ID":"3ede7ec3-5176-498b-890d-bc4614e18301","Type":"ContainerStarted","Data":"ed5c8351cfae9eb86e46d4559c935a34659e8f8212c94af8b92466e73f17ea00"}
Sep 09 23:49:09.420431 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.420408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pr48t" event={"ID":"3f909248-ff01-4fa8-a721-245433a608ef","Type":"ContainerStarted","Data":"ed821e37cb1789d811a5ed0199fe2589a4eff8ef8d87cc811c53f90e422f4765"}
Sep 09 23:49:09.422405 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.421944    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5qmv" event={"ID":"63c61f47-fa07-4006-abdb-e98e855f96a0","Type":"ContainerStarted","Data":"882c0b93b8c593c98f1b85db752da0229430b129330d99aedf9972189edd190a"}
Sep 09 23:49:09.431130 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.431084    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" podStartSLOduration=21.431072965 podStartE2EDuration="21.431072965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:52.39476754 +0000 UTC m=+5.715143605" watchObservedRunningTime="2026-09-09 23:49:09.431072965 +0000 UTC m=+22.751449029"
Sep 09 23:49:09.445052 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.445014    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pr48t" podStartSLOduration=11.44947961 podStartE2EDuration="22.445006326s" totalImagesPullingTime="10.995526716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:09.444689481 +0000 UTC m=+22.765065555" watchObservedRunningTime="2026-09-09 23:49:09.445006326 +0000 UTC m=+22.765382389"
Sep 09 23:49:09.475125 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.475086    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pmvmj" podStartSLOduration=3.891136606 podStartE2EDuration="22.475072299s" totalImagesPullingTime="18.583935693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:09.45967647 +0000 UTC m=+22.780052534" watchObservedRunningTime="2026-09-09 23:49:09.475072299 +0000 UTC m=+22.795448362"
Sep 09 23:49:09.475212 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.475190    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n8f6b" podStartSLOduration=3.8969977030000003 podStartE2EDuration="22.475184519s" totalImagesPullingTime="18.578186816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:09.475157055 +0000 UTC m=+22.795533118" watchObservedRunningTime="2026-09-09 23:49:09.475184519 +0000 UTC m=+22.795560582"
Sep 09 23:49:09.492972 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.492935    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5qmv" podStartSLOduration=3.845205535 podStartE2EDuration="22.492924208s" totalImagesPullingTime="18.647718673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:09.492668744 +0000 UTC m=+22.813044809" watchObservedRunningTime="2026-09-09 23:49:09.492924208 +0000 UTC m=+22.813300268"
Sep 09 23:49:09.507676 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:09.507641    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hfzpf" podStartSLOduration=3.930073683 podStartE2EDuration="22.507626077s" totalImagesPullingTime="18.577552394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:09.507124881 +0000 UTC m=+22.827500952" watchObservedRunningTime="2026-09-09 23:49:09.507626077 +0000 UTC m=+22.828002147"
Sep 09 23:49:10.196473 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.196285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:10.196686 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:10.196509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:10.432692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.432662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"0ba6e58c5ad74c06e640fdc5c4c859832c19534ca03559b4824b8edf5eecdcf3"}
Sep 09 23:49:10.432834 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.432770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"028a30be653d19759782022a551bdf0a4a3eca711e8bb648f177698a9b80f8e5"}
Sep 09 23:49:10.432834 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.432806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"9138806a7d08da8b09faa8b31d4ecfa387c73ff8183c0c214efeeaf1fd67f402"}
Sep 09 23:49:10.434482 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.434454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lj8x5" event={"ID":"608f8332-85df-464d-bc29-23ed75a98c0a","Type":"ContainerStarted","Data":"0b9d296df22d3ae1bc3d1d6add9d778429741ed3940b7d38f47f88b03c6b995c"}
Sep 09 23:49:10.449964 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.449855    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lj8x5" podStartSLOduration=4.863392901 podStartE2EDuration="23.449838328s" totalImagesPullingTime="18.586445427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:10.449583855 +0000 UTC m=+23.769959920" watchObservedRunningTime="2026-09-09 23:49:10.449838328 +0000 UTC m=+23.770214393"
Sep 09 23:49:10.473720 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:10.473683    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 23:49:11.168152 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.167836    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:49:10.473704719Z","UUID":"071f70f2-cd85-4512-af67-fd9af71d1dee","Handler":null,"Name":"","Endpoint":""}
Sep 09 23:49:11.169628 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.169607    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 23:49:11.169731 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.169639    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 23:49:11.196249 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.196224    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:11.196385 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.196264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:11.196385 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:11.196357    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:11.196681 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:11.196655    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:11.439042 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.438966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c" event={"ID":"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa","Type":"ContainerStarted","Data":"4e31f72a49a9fe50141c66ea49917cb673bb6eaf1d221a3b2024cdb6b5eec11a"}
Sep 09 23:49:11.659102 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.659072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:49:11.659775 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:11.659755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:49:12.196487 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.196460    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:12.196652 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:12.196570    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:12.445217 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.445175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"c82c5b891a76dcfe9db43bc85166e323d3cf30d03fdf9f037cfbee7226723ac1"}
Sep 09 23:49:12.448677 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.448604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c" event={"ID":"75aeaac5-084e-4991-bb7f-3c60d9f3bbaa","Type":"ContainerStarted","Data":"d08b7ed593af9db9670430a4e96ec7699544b0518dbf96ab9e43cab273511376"}
Sep 09 23:49:12.465823 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.465781    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4xt9c" podStartSLOduration=4.542045069 podStartE2EDuration="25.465767174s" totalImagesPullingTime="20.923722105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:12.464512203 +0000 UTC m=+25.784888268" watchObservedRunningTime="2026-09-09 23:49:12.465767174 +0000 UTC m=+25.786143237"
Sep 09 23:49:12.807601 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.807514    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:49:12.808195 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:12.808099    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pr48t"
Sep 09 23:49:13.196087 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:13.196058    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:13.196280 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:13.196195    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:13.196280 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:13.196252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:13.196402 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:13.196348    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:14.196624 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:14.196354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:14.197101 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:14.196678    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:14.453974 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:14.453898    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="9b0a5ad4b58f8f8f496d3f09567737ba766360552702681df595998cd87c2ef9" exitCode=0
Sep 09 23:49:14.454110 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:14.453988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"9b0a5ad4b58f8f8f496d3f09567737ba766360552702681df595998cd87c2ef9"}
Sep 09 23:49:14.458091 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:14.458057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" event={"ID":"402bf08e-c18b-4a14-ace4-6c259ab7aa8c","Type":"ContainerStarted","Data":"b1fdea61b3cc876069528cfaa9294e86822c96f4960f9fdc11b51f2202b860f4"}
Sep 09 23:49:14.504957 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:14.504916    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-st67z" podStartSLOduration=8.870987115 podStartE2EDuration="27.504902143s" totalImagesPullingTime="18.633915028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:14.503444842 +0000 UTC m=+27.823820906" watchObservedRunningTime="2026-09-09 23:49:14.504902143 +0000 UTC m=+27.825278207"
Sep 09 23:49:15.196124 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.196088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:15.196124 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.196119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:15.196385 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:15.196257    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:15.196757 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:15.196705    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:15.464431 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.461419    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:15.464431 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.461557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:15.464431 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.461630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:15.477114 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.477077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:15.477357 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.477334    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:15.909891 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.909695    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7j6c"]
Sep 09 23:49:15.910065 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.909984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:15.910120 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:15.910080    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:15.912461 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.912430    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ttqwg"]
Sep 09 23:49:15.912587 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.912533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:15.912698 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:15.912665    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:15.920983 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.920960    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2prt6"]
Sep 09 23:49:15.921100 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:15.921064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:15.921219 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:15.921191    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:16.464746 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:16.464716    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="708c3c2dc74f332f101187de3ecb1a69d95c9881ca4b92c2e38abe9cadc9865d" exitCode=0
Sep 09 23:49:16.465221 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:16.464800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"708c3c2dc74f332f101187de3ecb1a69d95c9881ca4b92c2e38abe9cadc9865d"}
Sep 09 23:49:17.198755 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:17.198282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:17.199313 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:17.199071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:18.196389 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:18.196357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:18.196807 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:18.196409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:18.196807 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:18.196495    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:18.196807 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:18.196615    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:18.471783 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:18.471751    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="e5175abec5964b54e2b8dabe3cc078fc286c93498ae6203c1380817537f6286d" exitCode=0
Sep 09 23:49:18.471929 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:18.471812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"e5175abec5964b54e2b8dabe3cc078fc286c93498ae6203c1380817537f6286d"}
Sep 09 23:49:19.196227 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:19.196195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:19.196399 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:19.196319    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:20.196337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:20.196309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:20.196782 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.196421    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2prt6" podUID="fc1b81b3-022d-4897-bf77-2c0b3b387446"
Sep 09 23:49:20.196782 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:20.196518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:20.196782 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.196664    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:49:20.860789 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:20.860751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:20.860805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:20.860837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860917    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860939    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860942    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:49:20.860970 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860963    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:49:20.861257 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860978    2576 projected.go:196] Error preparing data for projected volume kube-api-access-pc8c2 for pod openshift-network-diagnostics/network-check-target-m7j6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:20.861257 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.860993    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret podName:fc1b81b3-022d-4897-bf77-2c0b3b387446 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:52.860976158 +0000 UTC m=+66.181352214 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret") pod "global-pull-secret-syncer-2prt6" (UID: "fc1b81b3-022d-4897-bf77-2c0b3b387446") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:20.861257 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.861013    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:49:52.861002135 +0000 UTC m=+66.181378177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:20.861257 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:20.861027    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2 podName:6e6ead0a-efe0-4852-8fd9-04801d1eca1a nodeName:}" failed. No retries permitted until 2026-09-09 23:49:52.861018056 +0000 UTC m=+66.181394099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pc8c2" (UniqueName: "kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2") pod "network-check-target-m7j6c" (UID: "6e6ead0a-efe0-4852-8fd9-04801d1eca1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:21.196209 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:21.196132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:21.196367 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:21.196258    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7j6c" podUID="6e6ead0a-efe0-4852-8fd9-04801d1eca1a"
Sep 09 23:49:21.982842 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:21.982641    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeReady"
Sep 09 23:49:21.983015 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:21.982961    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 23:49:22.044699 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.044673    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jscps"]
Sep 09 23:49:22.050562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.050526    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lbz2f"]
Sep 09 23:49:22.050668 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.050655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.052907 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.052884    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 23:49:22.053030 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.052944    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4rs2\""
Sep 09 23:49:22.053093 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.053044    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 23:49:22.054309 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.054287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.056355 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.056334    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 23:49:22.059489 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.059434    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jscps"]
Sep 09 23:49:22.061129 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.060108    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 23:49:22.061129 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.060336    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 23:49:22.061129 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.060533    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqlfp\""
Sep 09 23:49:22.068235 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.067865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbz2f"]
Sep 09 23:49:22.169463 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.169643 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab711869-1ff0-4780-b836-8fac1eea468b-tmp-dir\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.169643 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86b9\" (UniqueName: \"kubernetes.io/projected/ab711869-1ff0-4780-b836-8fac1eea468b-kube-api-access-w86b9\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.169643 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgm9l\" (UniqueName: \"kubernetes.io/projected/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-kube-api-access-cgm9l\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.169643 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab711869-1ff0-4780-b836-8fac1eea468b-config-volume\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.169643 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.169641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-cert\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.196562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.196249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:22.196562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.196282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:22.199631 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.199610    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqch4\""
Sep 09 23:49:22.199841 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.199827    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 23:49:22.200031 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.200017    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 23:49:22.270743 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-cert\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.270743 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.270944 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab711869-1ff0-4780-b836-8fac1eea468b-tmp-dir\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.270944 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86b9\" (UniqueName: \"kubernetes.io/projected/ab711869-1ff0-4780-b836-8fac1eea468b-kube-api-access-w86b9\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.270944 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:22.270895    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:22.270944 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgm9l\" (UniqueName: \"kubernetes.io/projected/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-kube-api-access-cgm9l\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.271101 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.270947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab711869-1ff0-4780-b836-8fac1eea468b-config-volume\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.271101 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:22.270964    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:22.770943531 +0000 UTC m=+36.091319589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:22.271253 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.271234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab711869-1ff0-4780-b836-8fac1eea468b-tmp-dir\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.271458 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.271440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ab711869-1ff0-4780-b836-8fac1eea468b-config-volume\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.275271 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.275245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-cert\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.279335 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.279314    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgm9l\" (UniqueName: \"kubernetes.io/projected/5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4-kube-api-access-cgm9l\") pod \"ingress-canary-lbz2f\" (UID: \"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4\") " pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.279433 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.279338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86b9\" (UniqueName: \"kubernetes.io/projected/ab711869-1ff0-4780-b836-8fac1eea468b-kube-api-access-w86b9\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.371575 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.371528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lbz2f"
Sep 09 23:49:22.679872 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.679847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lbz2f"]
Sep 09 23:49:22.704320 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:49:22.704295    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2ce9b4_a71a_4589_b1b6_b70aaa2c90f4.slice/crio-7f8e3b4b8bdcc68a8a3e92d46652472e44eefa93642efd0e5701b3e6fc31f17e WatchSource:0}: Error finding container 7f8e3b4b8bdcc68a8a3e92d46652472e44eefa93642efd0e5701b3e6fc31f17e: Status 404 returned error can't find the container with id 7f8e3b4b8bdcc68a8a3e92d46652472e44eefa93642efd0e5701b3e6fc31f17e
Sep 09 23:49:22.773234 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:22.773213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:22.773333 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:22.773319    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:22.773379 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:22.773371    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:23.773357819 +0000 UTC m=+37.093733874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:23.195748 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.195713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:23.198439 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.198414    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 23:49:23.199134 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.198531    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 23:49:23.199423 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.199383    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-df57b\""
Sep 09 23:49:23.485574 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.485465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbz2f" event={"ID":"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4","Type":"ContainerStarted","Data":"7f8e3b4b8bdcc68a8a3e92d46652472e44eefa93642efd0e5701b3e6fc31f17e"}
Sep 09 23:49:23.489006 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.488979    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="121df9805efc3a8d7ca3686e107571ba32d389d2b4e059fe4c228706376381e3" exitCode=0
Sep 09 23:49:23.489110 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.489026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"121df9805efc3a8d7ca3686e107571ba32d389d2b4e059fe4c228706376381e3"}
Sep 09 23:49:23.781533 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:23.781451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:23.781704 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:23.781631    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:23.781754 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:23.781710    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:25.781686404 +0000 UTC m=+39.102062449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:24.497556 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:24.497503    2576 generic.go:356] "Generic (PLEG): container finished" podID="6bbacd63-8d59-408f-9ca0-b9b3b8993815" containerID="5842a31ef41da286ec33a0c9a065aec3910ff40a8a8d3dd639049cdf374b2d2b" exitCode=0
Sep 09 23:49:24.497966 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:24.497575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerDied","Data":"5842a31ef41da286ec33a0c9a065aec3910ff40a8a8d3dd639049cdf374b2d2b"}
Sep 09 23:49:25.501475 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:25.501308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lbz2f" event={"ID":"5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4","Type":"ContainerStarted","Data":"05b0d07b7e08a2ad207e7ac9e70a3520551fcbbb2fa158241258193278f288f6"}
Sep 09 23:49:25.504795 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:25.504771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" event={"ID":"6bbacd63-8d59-408f-9ca0-b9b3b8993815","Type":"ContainerStarted","Data":"ae8d55cf85edf9752b6d85b07302cf57743d7aad958d46ddc9b6344e02497c6f"}
Sep 09 23:49:25.516143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:25.516104    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lbz2f" podStartSLOduration=1.339380138 podStartE2EDuration="3.516090427s" totalImagesPullingTime="2.176710289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:25.515061815 +0000 UTC m=+38.835437879" watchObservedRunningTime="2026-09-09 23:49:25.516090427 +0000 UTC m=+38.836466490"
Sep 09 23:49:25.534465 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:25.534427    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bwtfb" podStartSLOduration=9.324475211 podStartE2EDuration="38.534414162s" totalImagesPullingTime="29.209938951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:25.532836904 +0000 UTC m=+38.853212968" watchObservedRunningTime="2026-09-09 23:49:25.534414162 +0000 UTC m=+38.854790223"
Sep 09 23:49:25.797189 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:25.797101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:25.797393 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:25.797224    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:25.797393 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:25.797281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:29.797267621 +0000 UTC m=+43.117643663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:29.822053 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:29.822014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:29.822525 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:29.822153    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:29.822525 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:29.822205    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:37.822192478 +0000 UTC m=+51.142568521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:37.873603 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:37.873570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:37.874118 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:37.873739    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:37.874118 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:37.873835    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:49:53.873812638 +0000 UTC m=+67.194188682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:47.486511 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:47.486481    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st67z"
Sep 09 23:49:52.869765 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.869723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:49:52.869765 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.869774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:52.870319 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.869803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:52.872163 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.872142    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 23:49:52.872294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.872146    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 23:49:52.872294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.872175    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 23:49:52.880272 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:52.880255    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:49:52.880397 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:52.880386    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:50:56.880368983 +0000 UTC m=+130.200745024 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : secret "metrics-daemon-secret" not found
Sep 09 23:49:52.882129 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.882112    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 23:49:52.883464 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.883435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fc1b81b3-022d-4897-bf77-2c0b3b387446-original-pull-secret\") pod \"global-pull-secret-syncer-2prt6\" (UID: \"fc1b81b3-022d-4897-bf77-2c0b3b387446\") " pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:52.892695 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.892675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8c2\" (UniqueName: \"kubernetes.io/projected/6e6ead0a-efe0-4852-8fd9-04801d1eca1a-kube-api-access-pc8c2\") pod \"network-check-target-m7j6c\" (UID: \"6e6ead0a-efe0-4852-8fd9-04801d1eca1a\") " pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:52.908681 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.908665    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-df57b\""
Sep 09 23:49:52.917376 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:52.917361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:53.107636 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.107604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2prt6"
Sep 09 23:49:53.186158 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.186134    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7j6c"]
Sep 09 23:49:53.189119 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:49:53.189097    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6ead0a_efe0_4852_8fd9_04801d1eca1a.slice/crio-6c75c7044e8b5da5caf621965465205e78b63f7990b8d8830c0b76b4cb16695a WatchSource:0}: Error finding container 6c75c7044e8b5da5caf621965465205e78b63f7990b8d8830c0b76b4cb16695a: Status 404 returned error can't find the container with id 6c75c7044e8b5da5caf621965465205e78b63f7990b8d8830c0b76b4cb16695a
Sep 09 23:49:53.374654 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.374630    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2prt6"]
Sep 09 23:49:53.377165 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:49:53.377139    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1b81b3_022d_4897_bf77_2c0b3b387446.slice/crio-46ae7e5a95490dff8522bd04fa6156269da762b64da1e0b16a37ae0963b36edd WatchSource:0}: Error finding container 46ae7e5a95490dff8522bd04fa6156269da762b64da1e0b16a37ae0963b36edd: Status 404 returned error can't find the container with id 46ae7e5a95490dff8522bd04fa6156269da762b64da1e0b16a37ae0963b36edd
Sep 09 23:49:53.570044 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.570011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2prt6" event={"ID":"fc1b81b3-022d-4897-bf77-2c0b3b387446","Type":"ContainerStarted","Data":"46ae7e5a95490dff8522bd04fa6156269da762b64da1e0b16a37ae0963b36edd"}
Sep 09 23:49:53.571143 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.571121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7j6c" event={"ID":"6e6ead0a-efe0-4852-8fd9-04801d1eca1a","Type":"ContainerStarted","Data":"6c75c7044e8b5da5caf621965465205e78b63f7990b8d8830c0b76b4cb16695a"}
Sep 09 23:49:53.876638 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:53.876534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:49:53.877058 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:53.876695    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:53.877058 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:49:53.876758    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:50:25.876739282 +0000 UTC m=+99.197115323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:58.589194 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:58.589155    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2prt6" event={"ID":"fc1b81b3-022d-4897-bf77-2c0b3b387446","Type":"ContainerStarted","Data":"9d15c9e2eafd8484dacf2b632d312f85c7a2ebbeca09c3d96f0601ddc18c35ef"}
Sep 09 23:49:58.590681 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:58.590654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7j6c" event={"ID":"6e6ead0a-efe0-4852-8fd9-04801d1eca1a","Type":"ContainerStarted","Data":"de4219ffe6d710c403a8b738215ce2618a4f7eb64255d4ea22399184eabf79ef"}
Sep 09 23:49:58.590789 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:58.590705    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:49:58.604960 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:58.604913    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2prt6" podStartSLOduration=67.314784419 podStartE2EDuration="1m11.6048826s" totalImagesPullingTime="4.290098181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:58.604377707 +0000 UTC m=+71.924753772" watchObservedRunningTime="2026-09-09 23:49:58.6048826 +0000 UTC m=+71.925258665"
Sep 09 23:49:58.617406 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:49:58.617355    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m7j6c" podStartSLOduration=67.143441032 podStartE2EDuration="1m11.617338705s" totalImagesPullingTime="4.473897673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:58.617060388 +0000 UTC m=+71.937436452" watchObservedRunningTime="2026-09-09 23:49:58.617338705 +0000 UTC m=+71.937714770"
Sep 09 23:50:25.975352 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:25.975325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:50:25.975777 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:50:25.975444    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:50:25.975777 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:50:25.975500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:51:29.97548576 +0000 UTC m=+163.295861801 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:50:26.745297 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.745266    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"]
Sep 09 23:50:26.749555 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.749524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.752364 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.752340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 23:50:26.752481 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.752346    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 23:50:26.752854 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.752835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 23:50:26.753739 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.753719    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 23:50:26.753918 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.753721    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 23:50:26.753918 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.753823    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 23:50:26.754086 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.753958    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 23:50:26.763035 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.763010    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"]
Sep 09 23:50:26.781177 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.781265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.781265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2257f87e-9a11-45fa-ae9e-a90c99044405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.781265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbvz\" (UniqueName: \"kubernetes.io/projected/2257f87e-9a11-45fa-ae9e-a90c99044405-kube-api-access-dmbvz\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.781373 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.781373 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.781284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882376 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882468 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882468 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2257f87e-9a11-45fa-ae9e-a90c99044405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882468 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbvz\" (UniqueName: \"kubernetes.io/projected/2257f87e-9a11-45fa-ae9e-a90c99044405-kube-api-access-dmbvz\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882599 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.882599 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.882594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.883151 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.883056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2257f87e-9a11-45fa-ae9e-a90c99044405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.885037 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.885018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-ca\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.885119 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.885071    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.885177 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.885159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.885274 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.885253    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2257f87e-9a11-45fa-ae9e-a90c99044405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:26.890528 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:26.890509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbvz\" (UniqueName: \"kubernetes.io/projected/2257f87e-9a11-45fa-ae9e-a90c99044405-kube-api-access-dmbvz\") pod \"cluster-proxy-proxy-agent-78fccfc548-jv7wd\" (UID: \"2257f87e-9a11-45fa-ae9e-a90c99044405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:27.073029 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:27.072999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:50:27.342062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:27.341989    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"]
Sep 09 23:50:27.346219 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:50:27.346192    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2257f87e_9a11_45fa_ae9e_a90c99044405.slice/crio-23170f14eba95098383a54d08cf87199a37871ab4800927dc105c1d67ed11498 WatchSource:0}: Error finding container 23170f14eba95098383a54d08cf87199a37871ab4800927dc105c1d67ed11498: Status 404 returned error can't find the container with id 23170f14eba95098383a54d08cf87199a37871ab4800927dc105c1d67ed11498
Sep 09 23:50:27.662411 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:27.662340    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerStarted","Data":"23170f14eba95098383a54d08cf87199a37871ab4800927dc105c1d67ed11498"}
Sep 09 23:50:29.595693 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:29.595661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m7j6c"
Sep 09 23:50:31.675320 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:31.675239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerStarted","Data":"f8cb34dac467032c9b63162b590af9ed62f48705e1d7827a4ae1cd7fe463d2cf"}
Sep 09 23:50:31.675320 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:31.675287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerStarted","Data":"54717ffe3cd8b92cb5048e5d993b50baf02f5b790792dc0173c1b86e4ff80906"}
Sep 09 23:50:31.675320 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:31.675306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerStarted","Data":"7618e3a6f97bac1dad7e3a1d27649e6720f9ad44cdd963a3a233e102cd89466e"}
Sep 09 23:50:31.692246 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:31.692204    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" podStartSLOduration=1.694101517 podStartE2EDuration="5.692190511s" totalImagesPullingTime="3.998088994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:50:31.69059466 +0000 UTC m=+105.010970723" watchObservedRunningTime="2026-09-09 23:50:31.692190511 +0000 UTC m=+105.012566575"
Sep 09 23:50:56.977553 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:50:56.977497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:50:56.978024 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:50:56.977643    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:50:56.978024 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:50:56.977704    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:52:58.977689026 +0000 UTC m=+252.298065071 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : secret "metrics-daemon-secret" not found
Sep 09 23:51:25.064646 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:51:25.064597    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jscps" podUID="ab711869-1ff0-4780-b836-8fac1eea468b"
Sep 09 23:51:25.213442 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:51:25.213404    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:51:25.822622 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:51:25.822596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jscps"
Sep 09 23:51:29.997842 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:51:29.997812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:51:29.998236 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:51:29.997946    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:51:29.998236 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:51:29.998015    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls podName:ab711869-1ff0-4780-b836-8fac1eea468b nodeName:}" failed. No retries permitted until 2026-09-09 23:53:31.99799502 +0000 UTC m=+285.318371064 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls") pod "dns-default-jscps" (UID: "ab711869-1ff0-4780-b836-8fac1eea468b") : secret "dns-default-metrics-tls" not found
Sep 09 23:51:40.195648 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:51:40.195574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:52:52.174171 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.174138    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-h5tx7"]
Sep 09 23:52:52.176869 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.176854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.179089 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.179067    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 23:52:52.179514 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.179491    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 23:52:52.179582 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.179565    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 23:52:52.179956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.179943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 23:52:52.180509 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.180494    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9p54c\""
Sep 09 23:52:52.185583 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.185534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 23:52:52.191452 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.191434    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-h5tx7"]
Sep 09 23:52:52.230931 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.230908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-snapshots\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.231073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.230934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.231073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.230969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.231073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.230987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-serving-cert\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.231073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.231026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29rf\" (UniqueName: \"kubernetes.io/projected/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-kube-api-access-j29rf\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.231223 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.231102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-tmp\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331631 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331631 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-serving-cert\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331849 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29rf\" (UniqueName: \"kubernetes.io/projected/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-kube-api-access-j29rf\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331849 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-tmp\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331849 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-snapshots\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.331961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.331862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.332157 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.332135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-tmp\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.332231 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.332216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.332493 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.332467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-snapshots\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.332630 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.332616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.334123 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.334107    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-serving-cert\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.340132 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.340112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29rf\" (UniqueName: \"kubernetes.io/projected/359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad-kube-api-access-j29rf\") pod \"insights-operator-76bcd6ccf5-h5tx7\" (UID: \"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad\") " pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.486221 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.486169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7"
Sep 09 23:52:52.751357 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:52.751289    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-h5tx7"]
Sep 09 23:52:52.754366 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:52:52.754340    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359a4c57_2c5f_40c5_a4a6_aa955cc6b8ad.slice/crio-f1e430438e252823e5c241ccef62c860b5f21aa42a1c13e3a98587d12983f207 WatchSource:0}: Error finding container f1e430438e252823e5c241ccef62c860b5f21aa42a1c13e3a98587d12983f207: Status 404 returned error can't find the container with id f1e430438e252823e5c241ccef62c860b5f21aa42a1c13e3a98587d12983f207
Sep 09 23:52:53.069214 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:53.069135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7" event={"ID":"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad","Type":"ContainerStarted","Data":"f1e430438e252823e5c241ccef62c860b5f21aa42a1c13e3a98587d12983f207"}
Sep 09 23:52:56.058319 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.058284    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-88sm9"]
Sep 09 23:52:56.061562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.061521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.064052 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.064030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2ktz8\""
Sep 09 23:52:56.064182 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.064030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 23:52:56.064337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.064321    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 23:52:56.071304 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.071281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88sm9"]
Sep 09 23:52:56.079631 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.079600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7" event={"ID":"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad","Type":"ContainerStarted","Data":"bdde3d3d788318eaf479351006304a6de717bdc94c745de00c1c6cfe2d157af1"}
Sep 09 23:52:56.110439 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.110399    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7" podStartSLOduration=1.735773001 podStartE2EDuration="4.110385829s" totalImagesPullingTime="2.374612828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:56.110022986 +0000 UTC m=+249.430399052" watchObservedRunningTime="2026-09-09 23:52:56.110385829 +0000 UTC m=+249.430761892"
Sep 09 23:52:56.157890 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.157855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.158024 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.157902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpzgg\" (UniqueName: \"kubernetes.io/projected/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-api-access-fpzgg\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.158024 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.157999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde60247-5f8b-4be1-874a-5286b0bee0bd-data-volume\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.158135 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.158022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.158135 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.158057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cde60247-5f8b-4be1-874a-5286b0bee0bd-crio-socket\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259336 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259472 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpzgg\" (UniqueName: \"kubernetes.io/projected/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-api-access-fpzgg\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259472 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde60247-5f8b-4be1-874a-5286b0bee0bd-data-volume\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259472 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259472 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.259444    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.259472 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cde60247-5f8b-4be1-874a-5286b0bee0bd-crio-socket\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259732 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.259519    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:56.759498918 +0000 UTC m=+250.079874985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.259732 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cde60247-5f8b-4be1-874a-5286b0bee0bd-crio-socket\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.259732 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.259595    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.259732 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.259661    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:56.75964313 +0000 UTC m=+250.080019189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.259732 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.259729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde60247-5f8b-4be1-874a-5286b0bee0bd-data-volume\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.269823 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.269800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpzgg\" (UniqueName: \"kubernetes.io/projected/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-api-access-fpzgg\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.762305 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.762254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.762490 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:56.762324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:56.762490 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.762398    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.762490 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.762413    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.762490 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.762450    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:57.762438116 +0000 UTC m=+251.082814158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.762490 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:56.762477    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:57.762464495 +0000 UTC m=+251.082840537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:57.769920 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:57.769878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:57.770397 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:57.769958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:57.770397 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:57.770021    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:57.770397 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:57.770062    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:57.770397 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:57.770077    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.770064006 +0000 UTC m=+253.090440048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:57.770397 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:57.770112    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.770098352 +0000 UTC m=+253.090474416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:58.012214 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:58.012187    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hfzpf_54d90c34-7f98-4289-99f4-32f0ecc6b581/dns-node-resolver/0.log"
Sep 09 23:52:58.979353 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:58.979317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:52:58.979758 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:58.979422    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:52:58.979758 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:58.979479    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs podName:7698041c-72bd-4509-840e-9add393b3bff nodeName:}" failed. No retries permitted until 2026-09-09 23:55:00.979461246 +0000 UTC m=+374.299837288 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs") pod "network-metrics-daemon-ttqwg" (UID: "7698041c-72bd-4509-840e-9add393b3bff") : secret "metrics-daemon-secret" not found
Sep 09 23:52:59.211816 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.211791    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n8f6b_3ede7ec3-5176-498b-890d-bc4614e18301/node-ca/0.log"
Sep 09 23:52:59.706251 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.706219    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-27mxf"]
Sep 09 23:52:59.709046 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.709030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.711522 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.711502    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 23:52:59.711821 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.711801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 23:52:59.711905 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.711805    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 23:52:59.712521 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.712508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-22qbl\""
Sep 09 23:52:59.712521 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.712518    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 23:52:59.718487 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.718468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-27mxf"]
Sep 09 23:52:59.785619 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.785592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:59.785619 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.785621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fv6\" (UniqueName: \"kubernetes.io/projected/d6195ca7-3d80-4f83-b270-6d20e255b741-kube-api-access-w8fv6\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.785783 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.785641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-key\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.785783 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:59.785740    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:59.785783 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.785768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:52:59.785903 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:59.785798    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:53:03.785783688 +0000 UTC m=+257.106159734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:59.785903 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:59.785816    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:59.785903 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.785825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-cabundle\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.785903 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:52:59.785860    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:53:03.785846513 +0000 UTC m=+257.106222575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:59.886849 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.886819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fv6\" (UniqueName: \"kubernetes.io/projected/d6195ca7-3d80-4f83-b270-6d20e255b741-kube-api-access-w8fv6\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.886849 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.886850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-key\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.887007 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.886984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-cabundle\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.887590 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.887575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-cabundle\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.889279 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.889263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d6195ca7-3d80-4f83-b270-6d20e255b741-signing-key\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:52:59.893958 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:52:59.893937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fv6\" (UniqueName: \"kubernetes.io/projected/d6195ca7-3d80-4f83-b270-6d20e255b741-kube-api-access-w8fv6\") pod \"service-ca-75d99c484-27mxf\" (UID: \"d6195ca7-3d80-4f83-b270-6d20e255b741\") " pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:53:00.012653 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:00.012597    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lbz2f_5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4/serve-healthcheck-canary/0.log"
Sep 09 23:53:00.018031 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:00.018012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-27mxf"
Sep 09 23:53:00.289284 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:00.289217    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-27mxf"]
Sep 09 23:53:00.292061 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:00.292039    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6195ca7_3d80_4f83_b270_6d20e255b741.slice/crio-9634c4bd71df2a67049b91b01d968802817102fb9ef0104e64ed9be8ed5c0b17 WatchSource:0}: Error finding container 9634c4bd71df2a67049b91b01d968802817102fb9ef0104e64ed9be8ed5c0b17: Status 404 returned error can't find the container with id 9634c4bd71df2a67049b91b01d968802817102fb9ef0104e64ed9be8ed5c0b17
Sep 09 23:53:01.095008 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:01.094969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-27mxf" event={"ID":"d6195ca7-3d80-4f83-b270-6d20e255b741","Type":"ContainerStarted","Data":"9634c4bd71df2a67049b91b01d968802817102fb9ef0104e64ed9be8ed5c0b17"}
Sep 09 23:53:03.102430 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:03.102393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-27mxf" event={"ID":"d6195ca7-3d80-4f83-b270-6d20e255b741","Type":"ContainerStarted","Data":"6d6d4a313adf0c2f01f32a1f03867f1051190ce15998ea2ffad7f157339547cb"}
Sep 09 23:53:03.117227 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:03.117181    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-27mxf" podStartSLOduration=2.340498684 podStartE2EDuration="4.117167424s" totalImagesPullingTime="1.77666874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:03.11666366 +0000 UTC m=+256.437039725" watchObservedRunningTime="2026-09-09 23:53:03.117167424 +0000 UTC m=+256.437543490"
Sep 09 23:53:03.816782 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:03.816744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:03.816984 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:03.816823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:03.816984 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:03.816922    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:03.816984 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:03.816936    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:53:03.817143 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:03.817007    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:53:11.816987573 +0000 UTC m=+265.137363618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:03.817143 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:03.817028    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:53:11.817018837 +0000 UTC m=+265.137394882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : configmap "kube-rbac-proxy" not found
Sep 09 23:53:11.874018 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:11.873923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:11.874018 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:11.874008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:11.874519 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:11.874128    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:11.874519 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:11.874210    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls podName:cde60247-5f8b-4be1-874a-5286b0bee0bd nodeName:}" failed. No retries permitted until 2026-09-09 23:53:27.874193335 +0000 UTC m=+281.194569378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-88sm9" (UID: "cde60247-5f8b-4be1-874a-5286b0bee0bd") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:11.874519 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:11.874439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cde60247-5f8b-4be1-874a-5286b0bee0bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:21.629819 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.629780    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd55f54d5-shmzb"]
Sep 09 23:53:21.632846 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.632830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.635401 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.635380    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 23:53:21.635644 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.635608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr4p2\""
Sep 09 23:53:21.635776 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.635642    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 23:53:21.636423 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.636408    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 23:53:21.640626 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.640608    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 23:53:21.645716 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.645696    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd55f54d5-shmzb"]
Sep 09 23:53:21.744391 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-trusted-ca\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-registry-tls\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pzg\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-kube-api-access-67pzg\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-registry-certificates\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407c45f9-a300-4a35-86be-f2497932860d-ca-trust-extracted\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744807 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-installation-pull-secrets\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744807 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-image-registry-private-configuration\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.744807 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.744685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-bound-sa-token\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.845972 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.845937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-registry-tls\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.845972 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.845970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pzg\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-kube-api-access-67pzg\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.845987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-registry-certificates\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407c45f9-a300-4a35-86be-f2497932860d-ca-trust-extracted\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-installation-pull-secrets\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-image-registry-private-configuration\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846496 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-bound-sa-token\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-trusted-ca\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407c45f9-a300-4a35-86be-f2497932860d-ca-trust-extracted\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.846964 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.846942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-registry-certificates\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.847826 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.847806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407c45f9-a300-4a35-86be-f2497932860d-trusted-ca\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.848660 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.848638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-registry-tls\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.848761 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.848673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-image-registry-private-configuration\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.848996 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.848975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407c45f9-a300-4a35-86be-f2497932860d-installation-pull-secrets\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.853281 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.853258    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pzg\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-kube-api-access-67pzg\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.853714 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.853694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407c45f9-a300-4a35-86be-f2497932860d-bound-sa-token\") pod \"image-registry-7cd55f54d5-shmzb\" (UID: \"407c45f9-a300-4a35-86be-f2497932860d\") " pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:21.943065 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:21.943000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:22.211023 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:22.210954    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd55f54d5-shmzb"]
Sep 09 23:53:22.213761 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:22.213735    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407c45f9_a300_4a35_86be_f2497932860d.slice/crio-ab2a3328b06bf6a7e34097f2867bae394aa15755fcdd2f92dac5b56ec74d3cfd WatchSource:0}: Error finding container ab2a3328b06bf6a7e34097f2867bae394aa15755fcdd2f92dac5b56ec74d3cfd: Status 404 returned error can't find the container with id ab2a3328b06bf6a7e34097f2867bae394aa15755fcdd2f92dac5b56ec74d3cfd
Sep 09 23:53:23.159639 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:23.159602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb" event={"ID":"407c45f9-a300-4a35-86be-f2497932860d","Type":"ContainerStarted","Data":"011c70126f83227c6d49e9553cc75c94a781285deddf29b5b7843313320f55dd"}
Sep 09 23:53:23.160047 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:23.159689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb" event={"ID":"407c45f9-a300-4a35-86be-f2497932860d","Type":"ContainerStarted","Data":"ab2a3328b06bf6a7e34097f2867bae394aa15755fcdd2f92dac5b56ec74d3cfd"}
Sep 09 23:53:23.160047 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:23.159732    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:23.177822 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:23.177778    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb" podStartSLOduration=2.177765736 podStartE2EDuration="2.177765736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:23.17703757 +0000 UTC m=+276.497413633" watchObservedRunningTime="2026-09-09 23:53:23.177765736 +0000 UTC m=+276.498141800"
Sep 09 23:53:27.074798 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:27.074754    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" podUID="2257f87e-9a11-45fa-ae9e-a90c99044405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 23:53:27.891482 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:27.891442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:27.893916 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:27.893885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cde60247-5f8b-4be1-874a-5286b0bee0bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-88sm9\" (UID: \"cde60247-5f8b-4be1-874a-5286b0bee0bd\") " pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:28.172621 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:28.172523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-88sm9"
Sep 09 23:53:28.442281 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:28.442212    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-88sm9"]
Sep 09 23:53:28.446325 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:28.446300    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde60247_5f8b_4be1_874a_5286b0bee0bd.slice/crio-0c54b13f9303c606de6b0968fd336d76664213a120f0c9c5bf0f364b34cfbc4d WatchSource:0}: Error finding container 0c54b13f9303c606de6b0968fd336d76664213a120f0c9c5bf0f364b34cfbc4d: Status 404 returned error can't find the container with id 0c54b13f9303c606de6b0968fd336d76664213a120f0c9c5bf0f364b34cfbc4d
Sep 09 23:53:28.823374 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:28.823337    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jscps" podUID="ab711869-1ff0-4780-b836-8fac1eea468b"
Sep 09 23:53:29.178986 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:29.178962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88sm9" event={"ID":"cde60247-5f8b-4be1-874a-5286b0bee0bd","Type":"ContainerStarted","Data":"8ca08b2a88eccae27cc1b2d9aee90c89a79bbf7c7d4bfffe93d5160ec1b37a3a"}
Sep 09 23:53:29.179257 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:29.179016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88sm9" event={"ID":"cde60247-5f8b-4be1-874a-5286b0bee0bd","Type":"ContainerStarted","Data":"0c54b13f9303c606de6b0968fd336d76664213a120f0c9c5bf0f364b34cfbc4d"}
Sep 09 23:53:29.179257 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:29.179110    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:30.184434 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:30.184398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88sm9" event={"ID":"cde60247-5f8b-4be1-874a-5286b0bee0bd","Type":"ContainerStarted","Data":"3f6af2b0dbf984e8a91a535dd97b5be0fd6c158052816859a264c55cdc745c48"}
Sep 09 23:53:31.189610 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:31.189577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-88sm9" event={"ID":"cde60247-5f8b-4be1-874a-5286b0bee0bd","Type":"ContainerStarted","Data":"d3fff17ff2eba0b3c494a00ecb6129a26ecb78cdd38d7deccc5f1c723f5c1001"}
Sep 09 23:53:31.206450 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:31.206408    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-88sm9" podStartSLOduration=33.2089756 podStartE2EDuration="35.206393255s" totalImagesPullingTime="1.997417655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:52:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:31.204596547 +0000 UTC m=+284.524972613" watchObservedRunningTime="2026-09-09 23:53:31.206393255 +0000 UTC m=+284.526769318"
Sep 09 23:53:32.020576 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:32.020518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:32.022969 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:32.022944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ab711869-1ff0-4780-b836-8fac1eea468b-metrics-tls\") pod \"dns-default-jscps\" (UID: \"ab711869-1ff0-4780-b836-8fac1eea468b\") " pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:32.182422 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:32.182399    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4rs2\""
Sep 09 23:53:32.190272 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:32.190256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:32.454839 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:32.454819    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jscps"]
Sep 09 23:53:32.456587 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:32.456568    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab711869_1ff0_4780_b836_8fac1eea468b.slice/crio-42c8824220a7c87cb967edd78ad5359ddc0ee24f61e2a23991719d1a66a94d04 WatchSource:0}: Error finding container 42c8824220a7c87cb967edd78ad5359ddc0ee24f61e2a23991719d1a66a94d04: Status 404 returned error can't find the container with id 42c8824220a7c87cb967edd78ad5359ddc0ee24f61e2a23991719d1a66a94d04
Sep 09 23:53:33.200530 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:33.200497    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jscps" event={"ID":"ab711869-1ff0-4780-b836-8fac1eea468b","Type":"ContainerStarted","Data":"42c8824220a7c87cb967edd78ad5359ddc0ee24f61e2a23991719d1a66a94d04"}
Sep 09 23:53:34.206350 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:34.206271    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jscps" event={"ID":"ab711869-1ff0-4780-b836-8fac1eea468b","Type":"ContainerStarted","Data":"64ea2313c6ae6fa259befd1965db1a335ee6e8b99405f4321b69909a4bf2eace"}
Sep 09 23:53:34.206350 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:34.206341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jscps" event={"ID":"ab711869-1ff0-4780-b836-8fac1eea468b","Type":"ContainerStarted","Data":"de21e1df7f90d83a1feb34014c8bcc568e8dd22ec8f53124e955b6a9099e5c2c"}
Sep 09 23:53:34.206797 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:34.206431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:34.222027 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:34.221980    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jscps" podStartSLOduration=250.819199166 podStartE2EDuration="4m12.221965237s" totalImagesPullingTime="1.402766071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:34.221795049 +0000 UTC m=+287.542171125" watchObservedRunningTime="2026-09-09 23:53:34.221965237 +0000 UTC m=+287.542341303"
Sep 09 23:53:35.101465 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.101426    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vv4hg"]
Sep 09 23:53:35.106377 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.106353    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.108944 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.108915    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 23:53:35.109101 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.109009    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 23:53:35.109101 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.109066    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 23:53:35.109222 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.109145    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 23:53:35.109222 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.109211    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgm5n\""
Sep 09 23:53:35.110179 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.110043    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 23:53:35.110462 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.110430    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 23:53:35.241184 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-wtmp\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnj2\" (UniqueName: \"kubernetes.io/projected/7e592fe9-4e63-4417-a036-85e94f80602a-kube-api-access-5dnj2\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-metrics-client-ca\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-sys\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-root\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.241562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.241462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-textfile\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342661 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-textfile\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342773 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-wtmp\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342847 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnj2\" (UniqueName: \"kubernetes.io/projected/7e592fe9-4e63-4417-a036-85e94f80602a-kube-api-access-5dnj2\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342907 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-metrics-client-ca\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342907 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-wtmp\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.342907 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-sys\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.342999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-sys\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.343030    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-textfile\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343298 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.343065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-root\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343298 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.343127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e592fe9-4e63-4417-a036-85e94f80602a-root\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343298 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:35.343207    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 23:53:35.343298 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:35.343259    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls podName:7e592fe9-4e63-4417-a036-85e94f80602a nodeName:}" failed. No retries permitted until 2026-09-09 23:53:35.843241065 +0000 UTC m=+289.163617120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls") pod "node-exporter-vv4hg" (UID: "7e592fe9-4e63-4417-a036-85e94f80602a") : secret "node-exporter-tls" not found
Sep 09 23:53:35.343510 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.343369    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-metrics-client-ca\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.343510 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.343501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.345572 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.345554    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.355752 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.355702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnj2\" (UniqueName: \"kubernetes.io/projected/7e592fe9-4e63-4417-a036-85e94f80602a-kube-api-access-5dnj2\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.847198 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.847164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:35.849490 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:35.849473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e592fe9-4e63-4417-a036-85e94f80602a-node-exporter-tls\") pod \"node-exporter-vv4hg\" (UID: \"7e592fe9-4e63-4417-a036-85e94f80602a\") " pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:36.016385 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:36.016354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vv4hg"
Sep 09 23:53:36.029445 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:36.029421    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e592fe9_4e63_4417_a036_85e94f80602a.slice/crio-ea3f784fa15aa676b687fc6c8d51a3f66257b1250fa2784dfed8b2910dc65c45 WatchSource:0}: Error finding container ea3f784fa15aa676b687fc6c8d51a3f66257b1250fa2784dfed8b2910dc65c45: Status 404 returned error can't find the container with id ea3f784fa15aa676b687fc6c8d51a3f66257b1250fa2784dfed8b2910dc65c45
Sep 09 23:53:36.213268 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:36.213173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv4hg" event={"ID":"7e592fe9-4e63-4417-a036-85e94f80602a","Type":"ContainerStarted","Data":"ea3f784fa15aa676b687fc6c8d51a3f66257b1250fa2784dfed8b2910dc65c45"}
Sep 09 23:53:37.074605 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:37.074577    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" podUID="2257f87e-9a11-45fa-ae9e-a90c99044405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 23:53:37.217695 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:37.217665    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e592fe9-4e63-4417-a036-85e94f80602a" containerID="75e8164ee20c7d3ae763b496615af8476de7c852f363f1d0a0d392e22a1e6205" exitCode=0
Sep 09 23:53:37.217809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:37.217715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv4hg" event={"ID":"7e592fe9-4e63-4417-a036-85e94f80602a","Type":"ContainerDied","Data":"75e8164ee20c7d3ae763b496615af8476de7c852f363f1d0a0d392e22a1e6205"}
Sep 09 23:53:38.223870 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:38.223838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv4hg" event={"ID":"7e592fe9-4e63-4417-a036-85e94f80602a","Type":"ContainerStarted","Data":"7b3f69b41f6de7c83e95d58507b1ac26543e3314162e7d294b51af8c071046f5"}
Sep 09 23:53:38.224415 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:38.223883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vv4hg" event={"ID":"7e592fe9-4e63-4417-a036-85e94f80602a","Type":"ContainerStarted","Data":"bd1cf517696806ebc1bbbf26d0518e881b1c75598615bb8da2be838f0dfec112"}
Sep 09 23:53:38.245422 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:38.245375    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vv4hg" podStartSLOduration=2.169542608 podStartE2EDuration="3.245361451s" totalImagesPullingTime="1.007783754s" totalInitContainerRuntime="68.035089ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:38.243475292 +0000 UTC m=+291.563851369" watchObservedRunningTime="2026-09-09 23:53:38.245361451 +0000 UTC m=+291.565737515"
Sep 09 23:53:39.505192 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.505157    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"]
Sep 09 23:53:39.508414 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.508398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.511013 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.510991    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 23:53:39.511139 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.511035    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 23:53:39.512049 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.512032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-78td1kv066u55\""
Sep 09 23:53:39.512144 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.512032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 23:53:39.512144 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.512066    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gsb2j\""
Sep 09 23:53:39.512144 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.512033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 23:53:39.517133 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.517116    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"]
Sep 09 23:53:39.675275 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebffc26b-9663-4ffc-a222-f1dc4281e944-audit-log\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675275 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-metrics-server-audit-profiles\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-client-certs\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-tls\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-client-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.675492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.675471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf5j\" (UniqueName: \"kubernetes.io/projected/ebffc26b-9663-4ffc-a222-f1dc4281e944-kube-api-access-dtf5j\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebffc26b-9663-4ffc-a222-f1dc4281e944-audit-log\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-metrics-server-audit-profiles\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-client-certs\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776602 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-tls\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-client-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.776961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf5j\" (UniqueName: \"kubernetes.io/projected/ebffc26b-9663-4ffc-a222-f1dc4281e944-kube-api-access-dtf5j\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.777065 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.776966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ebffc26b-9663-4ffc-a222-f1dc4281e944-audit-log\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.777383 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.777357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.777585 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.777567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ebffc26b-9663-4ffc-a222-f1dc4281e944-metrics-server-audit-profiles\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.779215 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.779194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-tls\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.779691 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.779666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-client-ca-bundle\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.779858 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.779834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ebffc26b-9663-4ffc-a222-f1dc4281e944-secret-metrics-server-client-certs\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.784905 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.784886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf5j\" (UniqueName: \"kubernetes.io/projected/ebffc26b-9663-4ffc-a222-f1dc4281e944-kube-api-access-dtf5j\") pod \"metrics-server-7cf6b58cb7-clcrc\" (UID: \"ebffc26b-9663-4ffc-a222-f1dc4281e944\") " pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.817837 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.817804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:39.870510 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.870478    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"]
Sep 09 23:53:39.874267 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.874249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:39.877397 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.876976    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wq9hd\""
Sep 09 23:53:39.877397 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.876996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 09 23:53:39.882253 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.882232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"]
Sep 09 23:53:39.978799 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.978768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:39.978956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:39.978827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22qb\" (UniqueName: \"kubernetes.io/projected/15282bc1-9142-437f-9c46-69c50ee15bd5-kube-api-access-k22qb\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.079354 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.079322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.079492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.079393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22qb\" (UniqueName: \"kubernetes.io/projected/15282bc1-9142-437f-9c46-69c50ee15bd5-kube-api-access-k22qb\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.079492 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:40.079480    2576 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 09 23:53:40.079607 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:40.079570    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert podName:15282bc1-9142-437f-9c46-69c50ee15bd5 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:40.579527131 +0000 UTC m=+293.899903174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-l6cvz" (UID: "15282bc1-9142-437f-9c46-69c50ee15bd5") : secret "monitoring-plugin-cert" not found
Sep 09 23:53:40.089767 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.089742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22qb\" (UniqueName: \"kubernetes.io/projected/15282bc1-9142-437f-9c46-69c50ee15bd5-kube-api-access-k22qb\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.092041 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.092020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"]
Sep 09 23:53:40.094580 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:40.094556    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebffc26b_9663_4ffc_a222_f1dc4281e944.slice/crio-e3af96dc61b7fa9d1b18f4b29f65fdec0320d4dfb4c757d92929a5f37f877f6e WatchSource:0}: Error finding container e3af96dc61b7fa9d1b18f4b29f65fdec0320d4dfb4c757d92929a5f37f877f6e: Status 404 returned error can't find the container with id e3af96dc61b7fa9d1b18f4b29f65fdec0320d4dfb4c757d92929a5f37f877f6e
Sep 09 23:53:40.231257 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.231221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc" event={"ID":"ebffc26b-9663-4ffc-a222-f1dc4281e944","Type":"ContainerStarted","Data":"e3af96dc61b7fa9d1b18f4b29f65fdec0320d4dfb4c757d92929a5f37f877f6e"}
Sep 09 23:53:40.583858 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.583828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.586292 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.586273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15282bc1-9142-437f-9c46-69c50ee15bd5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-l6cvz\" (UID: \"15282bc1-9142-437f-9c46-69c50ee15bd5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:40.787669 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:40.787637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:41.063715 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:41.063662    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"]
Sep 09 23:53:41.066838 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:41.066812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15282bc1_9142_437f_9c46_69c50ee15bd5.slice/crio-e2ded164aa8d28406671b1ee1d728a60b77e92e28be51abe3bf939739e39fe86 WatchSource:0}: Error finding container e2ded164aa8d28406671b1ee1d728a60b77e92e28be51abe3bf939739e39fe86: Status 404 returned error can't find the container with id e2ded164aa8d28406671b1ee1d728a60b77e92e28be51abe3bf939739e39fe86
Sep 09 23:53:41.236890 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:41.236797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz" event={"ID":"15282bc1-9142-437f-9c46-69c50ee15bd5","Type":"ContainerStarted","Data":"e2ded164aa8d28406671b1ee1d728a60b77e92e28be51abe3bf939739e39fe86"}
Sep 09 23:53:41.949513 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:41.949471    2576 patch_prober.go:28] interesting pod/image-registry-7cd55f54d5-shmzb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 23:53:41.950006 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:41.949556    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb" podUID="407c45f9-a300-4a35-86be-f2497932860d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 23:53:42.242045 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:42.241935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc" event={"ID":"ebffc26b-9663-4ffc-a222-f1dc4281e944","Type":"ContainerStarted","Data":"58a2c9cc01e093480115dfd5a0ad0c51ae70c9d74ff9cc63bdfc9c1975390167"}
Sep 09 23:53:42.259231 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:42.259088    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc" podStartSLOduration=1.654511486 podStartE2EDuration="3.259070002s" totalImagesPullingTime="1.604558516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:42.258919529 +0000 UTC m=+295.579295595" watchObservedRunningTime="2026-09-09 23:53:42.259070002 +0000 UTC m=+295.579446067"
Sep 09 23:53:43.196999 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:53:43.196960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ttqwg" podUID="7698041c-72bd-4509-840e-9add393b3bff"
Sep 09 23:53:43.246197 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:43.246164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz" event={"ID":"15282bc1-9142-437f-9c46-69c50ee15bd5","Type":"ContainerStarted","Data":"34f5af11f5f37f693bceb40e83a4ea6991b89b6b17e08d3e152bc9af332f05fa"}
Sep 09 23:53:43.246372 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:43.246352    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:43.250955 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:43.250934    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz"
Sep 09 23:53:43.277223 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:43.277184    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-l6cvz" podStartSLOduration=2.692207103 podStartE2EDuration="4.277171415s" totalImagesPullingTime="1.584964312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:43.276703839 +0000 UTC m=+296.597079902" watchObservedRunningTime="2026-09-09 23:53:43.277171415 +0000 UTC m=+296.597547476"
Sep 09 23:53:44.166906 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:44.166877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cd55f54d5-shmzb"
Sep 09 23:53:44.210265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:44.210239    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jscps"
Sep 09 23:53:47.074046 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.074012    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" podUID="2257f87e-9a11-45fa-ae9e-a90c99044405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 23:53:47.074491 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.074089    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd"
Sep 09 23:53:47.074689 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.074654    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f8cb34dac467032c9b63162b590af9ed62f48705e1d7827a4ae1cd7fe463d2cf"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 23:53:47.074763 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.074716    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" podUID="2257f87e-9a11-45fa-ae9e-a90c99044405" containerName="service-proxy" containerID="cri-o://f8cb34dac467032c9b63162b590af9ed62f48705e1d7827a4ae1cd7fe463d2cf" gracePeriod=30
Sep 09 23:53:47.121753 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.121720    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 23:53:47.261564 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.261516    2576 generic.go:356] "Generic (PLEG): container finished" podID="2257f87e-9a11-45fa-ae9e-a90c99044405" containerID="f8cb34dac467032c9b63162b590af9ed62f48705e1d7827a4ae1cd7fe463d2cf" exitCode=2
Sep 09 23:53:47.261779 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.261570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerDied","Data":"f8cb34dac467032c9b63162b590af9ed62f48705e1d7827a4ae1cd7fe463d2cf"}
Sep 09 23:53:47.261779 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:47.261620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fccfc548-jv7wd" event={"ID":"2257f87e-9a11-45fa-ae9e-a90c99044405","Type":"ContainerStarted","Data":"81e3c657256574c9e6aef61dd213c7ade74c4a7f8e70c70df638ea9a4babf3a8"}
Sep 09 23:53:58.195667 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.195632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:53:58.438492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.438461    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:53:58.442760 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.442745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.445276 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.445244    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 23:53:58.445406 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.445393    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 23:53:58.445485 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.445440    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 23:53:58.446280 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.446234    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 23:53:58.446361 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.446299    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 23:53:58.446746 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.446731    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fl7jn\""
Sep 09 23:53:58.446746 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.446744    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 23:53:58.446852 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.446759    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 23:53:58.451056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.451032    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 23:53:58.451977 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.451956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:53:58.617739 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdr6\" (UniqueName: \"kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617856 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617988 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.617988 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.617919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718660 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718660 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdr6\" (UniqueName: \"kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.718883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.718768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.719316 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.719296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.719393 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.719373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.719443 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.719373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.719954 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.719933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.721360 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.721341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.721455 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.721443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.727036 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.727018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdr6\" (UniqueName: \"kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6\") pod \"console-56775bcfd8-td49n\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") " pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:58.752476 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:58.752454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:53:59.022738 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:59.022662    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:53:59.026587 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:53:59.026560    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841775d1_e09c_4413_b0f5_916c7a7a769e.slice/crio-8aa5ac29b70d740737a5a9feeb614c73ffbfa04317964e224066c2efb87af81f WatchSource:0}: Error finding container 8aa5ac29b70d740737a5a9feeb614c73ffbfa04317964e224066c2efb87af81f: Status 404 returned error can't find the container with id 8aa5ac29b70d740737a5a9feeb614c73ffbfa04317964e224066c2efb87af81f
Sep 09 23:53:59.029206 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:59.029190    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:53:59.300349 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:59.300272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56775bcfd8-td49n" event={"ID":"841775d1-e09c-4413-b0f5-916c7a7a769e","Type":"ContainerStarted","Data":"8aa5ac29b70d740737a5a9feeb614c73ffbfa04317964e224066c2efb87af81f"}
Sep 09 23:53:59.818509 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:59.818449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:53:59.818509 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:53:59.818518    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:54:02.313083 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:02.312997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56775bcfd8-td49n" event={"ID":"841775d1-e09c-4413-b0f5-916c7a7a769e","Type":"ContainerStarted","Data":"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"}
Sep 09 23:54:02.329198 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:02.329147    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56775bcfd8-td49n" podStartSLOduration=1.406015515 podStartE2EDuration="4.329127748s" totalImagesPullingTime="2.923112233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:54:02.328942213 +0000 UTC m=+315.649318276" watchObservedRunningTime="2026-09-09 23:54:02.329127748 +0000 UTC m=+315.649503813"
Sep 09 23:54:08.753151 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:08.753109    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:54:08.753574 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:08.753185    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:54:08.757850 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:08.757829    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:54:09.341042 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:09.341011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:54:11.344360 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:11.344322    2576 generic.go:356] "Generic (PLEG): container finished" podID="359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad" containerID="bdde3d3d788318eaf479351006304a6de717bdc94c745de00c1c6cfe2d157af1" exitCode=0
Sep 09 23:54:11.344748 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:11.344391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7" event={"ID":"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad","Type":"ContainerDied","Data":"bdde3d3d788318eaf479351006304a6de717bdc94c745de00c1c6cfe2d157af1"}
Sep 09 23:54:11.344748 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:11.344687    2576 scope.go:117] "RemoveContainer" containerID="bdde3d3d788318eaf479351006304a6de717bdc94c745de00c1c6cfe2d157af1"
Sep 09 23:54:12.350124 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:12.350092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-h5tx7" event={"ID":"359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad","Type":"ContainerStarted","Data":"6414f97376ad55be84e4bff38fefc382da03f645262e6fd05861e85c44e66315"}
Sep 09 23:54:19.823154 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:19.823117    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:54:19.826963 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:54:19.826937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cf6b58cb7-clcrc"
Sep 09 23:55:00.980633 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:00.980593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:55:00.982967 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:00.982947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7698041c-72bd-4509-840e-9add393b3bff-metrics-certs\") pod \"network-metrics-daemon-ttqwg\" (UID: \"7698041c-72bd-4509-840e-9add393b3bff\") " pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:55:01.201384 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:01.201351    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqch4\""
Sep 09 23:55:01.207951 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:01.207935    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ttqwg"
Sep 09 23:55:01.477073 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:01.477050    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ttqwg"]
Sep 09 23:55:01.479534 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:55:01.479508    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7698041c_72bd_4509_840e_9add393b3bff.slice/crio-26bf9f7c6d6d5fcec686759fa7fdf9e3cb94cca223600f67546fff1c7a2b98ca WatchSource:0}: Error finding container 26bf9f7c6d6d5fcec686759fa7fdf9e3cb94cca223600f67546fff1c7a2b98ca: Status 404 returned error can't find the container with id 26bf9f7c6d6d5fcec686759fa7fdf9e3cb94cca223600f67546fff1c7a2b98ca
Sep 09 23:55:01.515507 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:01.515446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ttqwg" event={"ID":"7698041c-72bd-4509-840e-9add393b3bff","Type":"ContainerStarted","Data":"26bf9f7c6d6d5fcec686759fa7fdf9e3cb94cca223600f67546fff1c7a2b98ca"}
Sep 09 23:55:03.524584 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:03.524552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ttqwg" event={"ID":"7698041c-72bd-4509-840e-9add393b3bff","Type":"ContainerStarted","Data":"88a16f3ad7a0130352969723706059e5d74fe9ec05fda17a02dc73c021b435e1"}
Sep 09 23:55:03.524956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:03.524606    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ttqwg" event={"ID":"7698041c-72bd-4509-840e-9add393b3bff","Type":"ContainerStarted","Data":"f604fe54f66302ea5a47634b815565abbfeaf245fbf303132c862c4d68773e0a"}
Sep 09 23:55:03.544644 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:03.544573    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ttqwg" podStartSLOduration=375.573640014 podStartE2EDuration="6m16.544525758s" totalImagesPullingTime="970.885744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:03.541937889 +0000 UTC m=+376.862313953" watchObservedRunningTime="2026-09-09 23:55:03.544525758 +0000 UTC m=+376.864901822"
Sep 09 23:55:29.743032 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:29.743000    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:55:35.404533 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.404499    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"]
Sep 09 23:55:35.408968 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.408952    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.411400 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.411375    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h5v4r\""
Sep 09 23:55:35.411832 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.411815    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 23:55:35.412193 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.412180    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 23:55:35.417279 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.417259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"]
Sep 09 23:55:35.523337 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.523300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.523515 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.523355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.523515 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.523441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247cd\" (UniqueName: \"kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.624752 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.624726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.624826 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.624766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247cd\" (UniqueName: \"kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.624826 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.624802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.625079 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.625063    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.625145 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.625126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.633300 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.633279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-247cd\" (UniqueName: \"kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.718363 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.718305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:35.986148 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:35.986124    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"]
Sep 09 23:55:35.988600 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:55:35.988576    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca625ac_1f64_4d42_9f9f_35ed4d471f61.slice/crio-dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1 WatchSource:0}: Error finding container dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1: Status 404 returned error can't find the container with id dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1
Sep 09 23:55:36.635100 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:36.635066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk" event={"ID":"6ca625ac-1f64-4d42-9f9f-35ed4d471f61","Type":"ContainerStarted","Data":"dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1"}
Sep 09 23:55:42.659249 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:42.659218    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerID="8f937bfff98f759f53ee4b49e648c7e5c6457a428b91216e15fbf1bcf0c181fa" exitCode=0
Sep 09 23:55:42.659658 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:42.659268    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk" event={"ID":"6ca625ac-1f64-4d42-9f9f-35ed4d471f61","Type":"ContainerDied","Data":"8f937bfff98f759f53ee4b49e648c7e5c6457a428b91216e15fbf1bcf0c181fa"}
Sep 09 23:55:45.671611 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:45.671575    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerID="c86e09980436d768e9ff8648ea0173af4f299be7fa950f302309fdca9f56c341" exitCode=0
Sep 09 23:55:45.671996 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:45.671624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk" event={"ID":"6ca625ac-1f64-4d42-9f9f-35ed4d471f61","Type":"ContainerDied","Data":"c86e09980436d768e9ff8648ea0173af4f299be7fa950f302309fdca9f56c341"}
Sep 09 23:55:52.698760 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:52.698720    2576 generic.go:356] "Generic (PLEG): container finished" podID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerID="03060067011e739c96fa51f98a20d5e08731e8d4f18e561594fc87faaa62e753" exitCode=0
Sep 09 23:55:52.699159 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:52.698804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk" event={"ID":"6ca625ac-1f64-4d42-9f9f-35ed4d471f61","Type":"ContainerDied","Data":"03060067011e739c96fa51f98a20d5e08731e8d4f18e561594fc87faaa62e753"}
Sep 09 23:55:53.841577 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.841524    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:53.873918 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.873888    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util\") pod \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") "
Sep 09 23:55:53.874070 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.873938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-247cd\" (UniqueName: \"kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd\") pod \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") "
Sep 09 23:55:53.874070 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.873994    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle\") pod \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\" (UID: \"6ca625ac-1f64-4d42-9f9f-35ed4d471f61\") "
Sep 09 23:55:53.874649 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.874622    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle" (OuterVolumeSpecName: "bundle") pod "6ca625ac-1f64-4d42-9f9f-35ed4d471f61" (UID: "6ca625ac-1f64-4d42-9f9f-35ed4d471f61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:55:53.876316 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.876296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd" (OuterVolumeSpecName: "kube-api-access-247cd") pod "6ca625ac-1f64-4d42-9f9f-35ed4d471f61" (UID: "6ca625ac-1f64-4d42-9f9f-35ed4d471f61"). InnerVolumeSpecName "kube-api-access-247cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:55:53.878424 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.878400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util" (OuterVolumeSpecName: "util") pod "6ca625ac-1f64-4d42-9f9f-35ed4d471f61" (UID: "6ca625ac-1f64-4d42-9f9f-35ed4d471f61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:55:53.974887 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.974811    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247cd\" (UniqueName: \"kubernetes.io/projected/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-kube-api-access-247cd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:53.974887 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.974838    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:53.974887 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:53.974850    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ca625ac-1f64-4d42-9f9f-35ed4d471f61-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:54.708050 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:54.708018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk"
Sep 09 23:55:54.708212 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:54.708013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406kvlk" event={"ID":"6ca625ac-1f64-4d42-9f9f-35ed4d471f61","Type":"ContainerDied","Data":"dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1"}
Sep 09 23:55:54.708212 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:54.708154    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc02be37777f6dbd40441a54b975fa9508831f4a743fe1748a342ad9cd15b4c1"
Sep 09 23:55:54.762649 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:54.762617    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56775bcfd8-td49n" podUID="841775d1-e09c-4413-b0f5-916c7a7a769e" containerName="console" containerID="cri-o://6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256" gracePeriod=15
Sep 09 23:55:55.033723 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.033696    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:55:55.084788 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084763    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.084921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084799    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzdr6\" (UniqueName: \"kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.084921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084832    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.084921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.084921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084864    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.084921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.085170 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.084949    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca\") pod \"841775d1-e09c-4413-b0f5-916c7a7a769e\" (UID: \"841775d1-e09c-4413-b0f5-916c7a7a769e\") "
Sep 09 23:55:55.085341 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.085284    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config" (OuterVolumeSpecName: "console-config") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:55.085341 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.085308    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:55.085479 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.085427    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca" (OuterVolumeSpecName: "service-ca") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:55.085513 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.085465    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:55.087324 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.087302    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6" (OuterVolumeSpecName: "kube-api-access-mzdr6") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "kube-api-access-mzdr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:55:55.087397 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.087305    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:55.087397 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.087327    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "841775d1-e09c-4413-b0f5-916c7a7a769e" (UID: "841775d1-e09c-4413-b0f5-916c7a7a769e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:55.185766 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185743    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-oauth-serving-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185766 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185762    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdr6\" (UniqueName: \"kubernetes.io/projected/841775d1-e09c-4413-b0f5-916c7a7a769e-kube-api-access-mzdr6\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185772    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-console-config\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185781    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-serving-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185789    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-trusted-ca-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185798    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/841775d1-e09c-4413-b0f5-916c7a7a769e-console-oauth-config\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.185883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.185805    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/841775d1-e09c-4413-b0f5-916c7a7a769e-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:55:55.712864 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.712827    2576 generic.go:356] "Generic (PLEG): container finished" podID="841775d1-e09c-4413-b0f5-916c7a7a769e" containerID="6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256" exitCode=0
Sep 09 23:55:55.713027 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.712870    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56775bcfd8-td49n"
Sep 09 23:55:55.713027 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.712884    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56775bcfd8-td49n" event={"ID":"841775d1-e09c-4413-b0f5-916c7a7a769e","Type":"ContainerDied","Data":"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"}
Sep 09 23:55:55.713027 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.712932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56775bcfd8-td49n" event={"ID":"841775d1-e09c-4413-b0f5-916c7a7a769e","Type":"ContainerDied","Data":"8aa5ac29b70d740737a5a9feeb614c73ffbfa04317964e224066c2efb87af81f"}
Sep 09 23:55:55.713027 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.712954    2576 scope.go:117] "RemoveContainer" containerID="6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"
Sep 09 23:55:55.728728 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.728700    2576 scope.go:117] "RemoveContainer" containerID="6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"
Sep 09 23:55:55.728999 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:55:55.728981    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256\": container with ID starting with 6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256 not found: ID does not exist" containerID="6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"
Sep 09 23:55:55.729050 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.729007    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256"} err="failed to get container status \"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256\": rpc error: code = NotFound desc = could not find container \"6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256\": container with ID starting with 6b605b0c533cdf0127886fcb394a923456dd7723bb717972dcc814aba0426256 not found: ID does not exist"
Sep 09 23:55:55.732441 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.732421    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:55:55.735809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:55.735788    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56775bcfd8-td49n"]
Sep 09 23:55:57.205839 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:57.205801    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841775d1-e09c-4413-b0f5-916c7a7a769e" path="/var/lib/kubelet/pods/841775d1-e09c-4413-b0f5-916c7a7a769e/volumes"
Sep 09 23:55:58.317799 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.317767    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"]
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318006    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="pull"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318019    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="pull"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318027    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="extract"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318032    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="extract"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318043    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841775d1-e09c-4413-b0f5-916c7a7a769e" containerName="console"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318049    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="841775d1-e09c-4413-b0f5-916c7a7a769e" containerName="console"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318054    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="util"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318061    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="util"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318114    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca625ac-1f64-4d42-9f9f-35ed4d471f61" containerName="extract"
Sep 09 23:55:58.318146 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.318121    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="841775d1-e09c-4413-b0f5-916c7a7a769e" containerName="console"
Sep 09 23:55:58.321245 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.321231    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.323700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.323680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:55:58.323700 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.323690    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 23:55:58.323892 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.323693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-j2hbr\""
Sep 09 23:55:58.336618 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.336599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"]
Sep 09 23:55:58.414203 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.414178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5zd\" (UniqueName: \"kubernetes.io/projected/2f35da12-4bef-437a-abd8-dcc319b21d17-kube-api-access-kp5zd\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.414341 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.414214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f35da12-4bef-437a-abd8-dcc319b21d17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.515473 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.515446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f35da12-4bef-437a-abd8-dcc319b21d17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.515569 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.515508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5zd\" (UniqueName: \"kubernetes.io/projected/2f35da12-4bef-437a-abd8-dcc319b21d17-kube-api-access-kp5zd\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.515843 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.515825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f35da12-4bef-437a-abd8-dcc319b21d17-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.523085 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.523065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5zd\" (UniqueName: \"kubernetes.io/projected/2f35da12-4bef-437a-abd8-dcc319b21d17-kube-api-access-kp5zd\") pod \"cert-manager-operator-controller-manager-55c444bf86-klz6t\" (UID: \"2f35da12-4bef-437a-abd8-dcc319b21d17\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.630086 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.630053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"
Sep 09 23:55:58.902255 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:58.902232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t"]
Sep 09 23:55:58.904692 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:55:58.904666    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f35da12_4bef_437a_abd8_dcc319b21d17.slice/crio-6f1db0f1308179a4bb5df9322d9205fd9213fe0b4212459f0363f7965b45fcd0 WatchSource:0}: Error finding container 6f1db0f1308179a4bb5df9322d9205fd9213fe0b4212459f0363f7965b45fcd0: Status 404 returned error can't find the container with id 6f1db0f1308179a4bb5df9322d9205fd9213fe0b4212459f0363f7965b45fcd0
Sep 09 23:55:59.730401 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:55:59.730364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t" event={"ID":"2f35da12-4bef-437a-abd8-dcc319b21d17","Type":"ContainerStarted","Data":"6f1db0f1308179a4bb5df9322d9205fd9213fe0b4212459f0363f7965b45fcd0"}
Sep 09 23:56:01.745058 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:01.744970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t" event={"ID":"2f35da12-4bef-437a-abd8-dcc319b21d17","Type":"ContainerStarted","Data":"0502c575dac364000cd7c9020e014853130a65db9a925c9bd5d6dbaffe99dac8"}
Sep 09 23:56:01.767000 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:01.766949    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-klz6t" podStartSLOduration=1.290825366 podStartE2EDuration="3.766934258s" totalImagesPullingTime="2.476108892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:01.765684706 +0000 UTC m=+435.086060770" watchObservedRunningTime="2026-09-09 23:56:01.766934258 +0000 UTC m=+435.087310322"
Sep 09 23:56:02.639752 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.639715    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"]
Sep 09 23:56:02.643128 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.643106    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.645524 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.645504    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 23:56:02.646492 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.646470    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h5v4r\""
Sep 09 23:56:02.646616 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.646500    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 23:56:02.651612 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.651583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"]
Sep 09 23:56:02.750989 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.750954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmj4\" (UniqueName: \"kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.750989 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.750994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.751419 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.751072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.851727 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.851699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.851901 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.851753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmj4\" (UniqueName: \"kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.851901 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.851773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.852079 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.852062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.852147 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.852130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.860556 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.860489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmj4\" (UniqueName: \"kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:02.952653 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:02.952587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:03.225360 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:03.225259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"]
Sep 09 23:56:03.228051 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:03.228015    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720a84cf_087b_47f5_b39a_6d04eccbe958.slice/crio-e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3 WatchSource:0}: Error finding container e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3: Status 404 returned error can't find the container with id e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3
Sep 09 23:56:03.756163 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:03.756129    2576 generic.go:356] "Generic (PLEG): container finished" podID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerID="47fd38ce2254de8647c018658de3b94fee9dc4b69e2f4737c94f28d91f83eb92" exitCode=0
Sep 09 23:56:03.756647 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:03.756200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b" event={"ID":"720a84cf-087b-47f5-b39a-6d04eccbe958","Type":"ContainerDied","Data":"47fd38ce2254de8647c018658de3b94fee9dc4b69e2f4737c94f28d91f83eb92"}
Sep 09 23:56:03.756647 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:03.756234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b" event={"ID":"720a84cf-087b-47f5-b39a-6d04eccbe958","Type":"ContainerStarted","Data":"e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3"}
Sep 09 23:56:05.672041 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.671926    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"]
Sep 09 23:56:05.682195 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.682173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.684793 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.684563    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 23:56:05.684793 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.684573    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4n6lh\""
Sep 09 23:56:05.685332 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.685304    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"]
Sep 09 23:56:05.685638 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.685612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 23:56:05.773034 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.773004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.773214 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.773044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97zc\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-kube-api-access-p97zc\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.874252 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.874221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.874403 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.874255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97zc\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-kube-api-access-p97zc\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.881752 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.881722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.881939 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.881920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97zc\" (UniqueName: \"kubernetes.io/projected/c7c95a5a-08dc-4853-945e-49d8f1c63ca1-kube-api-access-p97zc\") pod \"cert-manager-cainjector-559cdd8c87-8ldbd\" (UID: \"c7c95a5a-08dc-4853-945e-49d8f1c63ca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:05.992755 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:05.992731    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"
Sep 09 23:56:06.264588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:06.264565    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd"]
Sep 09 23:56:06.266726 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:06.266696    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7c95a5a_08dc_4853_945e_49d8f1c63ca1.slice/crio-9beb5b814587ee4dcddce446a57016c297f6083340b8e1393f5052b66df78c9c WatchSource:0}: Error finding container 9beb5b814587ee4dcddce446a57016c297f6083340b8e1393f5052b66df78c9c: Status 404 returned error can't find the container with id 9beb5b814587ee4dcddce446a57016c297f6083340b8e1393f5052b66df78c9c
Sep 09 23:56:06.770659 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:06.770625    2576 generic.go:356] "Generic (PLEG): container finished" podID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerID="46d0a9a5d8e8222ee36b28c51c32daaef8b8be98c9d371f88974731d2e8dacf4" exitCode=0
Sep 09 23:56:06.771062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:06.770710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b" event={"ID":"720a84cf-087b-47f5-b39a-6d04eccbe958","Type":"ContainerDied","Data":"46d0a9a5d8e8222ee36b28c51c32daaef8b8be98c9d371f88974731d2e8dacf4"}
Sep 09 23:56:06.771908 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:06.771878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd" event={"ID":"c7c95a5a-08dc-4853-945e-49d8f1c63ca1","Type":"ContainerStarted","Data":"9beb5b814587ee4dcddce446a57016c297f6083340b8e1393f5052b66df78c9c"}
Sep 09 23:56:07.779239 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:07.779203    2576 generic.go:356] "Generic (PLEG): container finished" podID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerID="7151ee6ad9679ef875797a75d505f43e69a41a7746df574ffdc45ab20bf37716" exitCode=0
Sep 09 23:56:07.779731 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:07.779269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b" event={"ID":"720a84cf-087b-47f5-b39a-6d04eccbe958","Type":"ContainerDied","Data":"7151ee6ad9679ef875797a75d505f43e69a41a7746df574ffdc45ab20bf37716"}
Sep 09 23:56:08.918914 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:08.918887    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:08.998208 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:08.998172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util\") pod \"720a84cf-087b-47f5-b39a-6d04eccbe958\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") "
Sep 09 23:56:08.998377 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:08.998262    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle\") pod \"720a84cf-087b-47f5-b39a-6d04eccbe958\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") "
Sep 09 23:56:08.998377 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:08.998297    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnmj4\" (UniqueName: \"kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4\") pod \"720a84cf-087b-47f5-b39a-6d04eccbe958\" (UID: \"720a84cf-087b-47f5-b39a-6d04eccbe958\") "
Sep 09 23:56:08.998732 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:08.998700    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle" (OuterVolumeSpecName: "bundle") pod "720a84cf-087b-47f5-b39a-6d04eccbe958" (UID: "720a84cf-087b-47f5-b39a-6d04eccbe958"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:09.000817 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.000795    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4" (OuterVolumeSpecName: "kube-api-access-fnmj4") pod "720a84cf-087b-47f5-b39a-6d04eccbe958" (UID: "720a84cf-087b-47f5-b39a-6d04eccbe958"). InnerVolumeSpecName "kube-api-access-fnmj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:56:09.003764 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.003740    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util" (OuterVolumeSpecName: "util") pod "720a84cf-087b-47f5-b39a-6d04eccbe958" (UID: "720a84cf-087b-47f5-b39a-6d04eccbe958"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:09.099218 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.099185    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:09.099218 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.099211    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnmj4\" (UniqueName: \"kubernetes.io/projected/720a84cf-087b-47f5-b39a-6d04eccbe958-kube-api-access-fnmj4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:09.099218 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.099220    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/720a84cf-087b-47f5-b39a-6d04eccbe958-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:09.795098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.795062    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b"
Sep 09 23:56:09.795248 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.795064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6cr9b" event={"ID":"720a84cf-087b-47f5-b39a-6d04eccbe958","Type":"ContainerDied","Data":"e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3"}
Sep 09 23:56:09.795248 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:09.795221    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01715d8ea1d49be2d256cfc9f6887c94cd9489e3743954f33ca5309d11a71f3"
Sep 09 23:56:10.801649 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:10.801555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd" event={"ID":"c7c95a5a-08dc-4853-945e-49d8f1c63ca1","Type":"ContainerStarted","Data":"871270320eab21d167f8c90f7509c19c0d350b68e059ab8ad95ebd7f1a720558"}
Sep 09 23:56:10.816191 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:10.816144    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8ldbd" podStartSLOduration=1.550751464 podStartE2EDuration="5.816129306s" totalImagesPullingTime="4.265377842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:10.815249629 +0000 UTC m=+444.135625693" watchObservedRunningTime="2026-09-09 23:56:10.816129306 +0000 UTC m=+444.136505369"
Sep 09 23:56:22.702513 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702477    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbjfs"]
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702735    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="util"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702746    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="util"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702755    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="pull"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702761    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="pull"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702766    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="extract"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702772    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="extract"
Sep 09 23:56:22.702956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.702812    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="720a84cf-087b-47f5-b39a-6d04eccbe958" containerName="extract"
Sep 09 23:56:22.754484 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.754453    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbjfs"]
Sep 09 23:56:22.754646 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.754557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:22.757305 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.757285    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-crg4r\""
Sep 09 23:56:22.894294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.894267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:22.894407 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.894315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jsf7\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-kube-api-access-4jsf7\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:22.994878 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.994819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jsf7\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-kube-api-access-4jsf7\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:22.994878 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:22.994865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:23.002385 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.002357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jsf7\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-kube-api-access-4jsf7\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:23.002473 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.002425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d86d64c-1fbc-4885-b30d-437dbdea3a83-bound-sa-token\") pod \"cert-manager-55b4d954d5-jbjfs\" (UID: \"9d86d64c-1fbc-4885-b30d-437dbdea3a83\") " pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:23.032956 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.032935    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"]
Sep 09 23:56:23.052376 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.052354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"]
Sep 09 23:56:23.052478 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.052442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.054821 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.054801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h5v4r\""
Sep 09 23:56:23.054930 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.054868    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 23:56:23.054930 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.054873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 23:56:23.063049 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.063032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jbjfs"
Sep 09 23:56:23.196218 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.196185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.196350 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.196226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.196350 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.196289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45rd\" (UniqueName: \"kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.296774 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.296717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.296774 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.296754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.296918 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.296785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45rd\" (UniqueName: \"kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.297066 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.297048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.297107 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.297076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.304745 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.304722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45rd\" (UniqueName: \"kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.361650 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.361630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:23.377346 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.377310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jbjfs"]
Sep 09 23:56:23.380096 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:23.380072    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d86d64c_1fbc_4885_b30d_437dbdea3a83.slice/crio-e31dbfcd808d161044315eb600ac5bbde7eff2a50df8c5872aed643c73b79c24 WatchSource:0}: Error finding container e31dbfcd808d161044315eb600ac5bbde7eff2a50df8c5872aed643c73b79c24: Status 404 returned error can't find the container with id e31dbfcd808d161044315eb600ac5bbde7eff2a50df8c5872aed643c73b79c24
Sep 09 23:56:23.662033 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.662009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"]
Sep 09 23:56:23.664331 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:23.664309    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310e75f9_ab8c_4c72_95d1_12022d833966.slice/crio-341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971 WatchSource:0}: Error finding container 341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971: Status 404 returned error can't find the container with id 341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971
Sep 09 23:56:23.853830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.853795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jbjfs" event={"ID":"9d86d64c-1fbc-4885-b30d-437dbdea3a83","Type":"ContainerStarted","Data":"bf520bbd5fd1caa552b166d6d035d713ab9233559d9ab1b5dace351f2e95110b"}
Sep 09 23:56:23.854251 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.853838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jbjfs" event={"ID":"9d86d64c-1fbc-4885-b30d-437dbdea3a83","Type":"ContainerStarted","Data":"e31dbfcd808d161044315eb600ac5bbde7eff2a50df8c5872aed643c73b79c24"}
Sep 09 23:56:23.855432 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.855404    2576 generic.go:356] "Generic (PLEG): container finished" podID="310e75f9-ab8c-4c72-95d1-12022d833966" containerID="0a5f92b2dc8b0ec00c718ab25020410170cbe079f28a92c302e88e9bc2069a52" exitCode=0
Sep 09 23:56:23.855514 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.855452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj" event={"ID":"310e75f9-ab8c-4c72-95d1-12022d833966","Type":"ContainerDied","Data":"0a5f92b2dc8b0ec00c718ab25020410170cbe079f28a92c302e88e9bc2069a52"}
Sep 09 23:56:23.855514 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.855482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj" event={"ID":"310e75f9-ab8c-4c72-95d1-12022d833966","Type":"ContainerStarted","Data":"341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971"}
Sep 09 23:56:23.870138 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:23.870094    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jbjfs" podStartSLOduration=1.870080278 podStartE2EDuration="1.870080278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:23.868502434 +0000 UTC m=+457.188878497" watchObservedRunningTime="2026-09-09 23:56:23.870080278 +0000 UTC m=+457.190456342"
Sep 09 23:56:24.861291 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:24.861259    2576 generic.go:356] "Generic (PLEG): container finished" podID="310e75f9-ab8c-4c72-95d1-12022d833966" containerID="d9d26bf9674734264677370308d08d7f696fff4e005cee6646e7de5979b98103" exitCode=0
Sep 09 23:56:24.861714 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:24.861357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj" event={"ID":"310e75f9-ab8c-4c72-95d1-12022d833966","Type":"ContainerDied","Data":"d9d26bf9674734264677370308d08d7f696fff4e005cee6646e7de5979b98103"}
Sep 09 23:56:25.868059 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:25.868022    2576 generic.go:356] "Generic (PLEG): container finished" podID="310e75f9-ab8c-4c72-95d1-12022d833966" containerID="f3da702f221d76c790822e961dd4ab36a75b79307d81383389eb1d4a00406df4" exitCode=0
Sep 09 23:56:25.868432 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:25.868076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj" event={"ID":"310e75f9-ab8c-4c72-95d1-12022d833966","Type":"ContainerDied","Data":"f3da702f221d76c790822e961dd4ab36a75b79307d81383389eb1d4a00406df4"}
Sep 09 23:56:26.988511 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:26.988490    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:27.129565 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.129459    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g45rd\" (UniqueName: \"kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd\") pod \"310e75f9-ab8c-4c72-95d1-12022d833966\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") "
Sep 09 23:56:27.129565 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.129507    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle\") pod \"310e75f9-ab8c-4c72-95d1-12022d833966\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") "
Sep 09 23:56:27.129765 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.129571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util\") pod \"310e75f9-ab8c-4c72-95d1-12022d833966\" (UID: \"310e75f9-ab8c-4c72-95d1-12022d833966\") "
Sep 09 23:56:27.130155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.130126    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle" (OuterVolumeSpecName: "bundle") pod "310e75f9-ab8c-4c72-95d1-12022d833966" (UID: "310e75f9-ab8c-4c72-95d1-12022d833966"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:27.131588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.131566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd" (OuterVolumeSpecName: "kube-api-access-g45rd") pod "310e75f9-ab8c-4c72-95d1-12022d833966" (UID: "310e75f9-ab8c-4c72-95d1-12022d833966"). InnerVolumeSpecName "kube-api-access-g45rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:56:27.137443 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.137419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util" (OuterVolumeSpecName: "util") pod "310e75f9-ab8c-4c72-95d1-12022d833966" (UID: "310e75f9-ab8c-4c72-95d1-12022d833966"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:27.230160 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.230132    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:27.230160 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.230155    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/310e75f9-ab8c-4c72-95d1-12022d833966-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:27.230288 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.230164    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45rd\" (UniqueName: \"kubernetes.io/projected/310e75f9-ab8c-4c72-95d1-12022d833966-kube-api-access-g45rd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:27.878451 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.878413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj" event={"ID":"310e75f9-ab8c-4c72-95d1-12022d833966","Type":"ContainerDied","Data":"341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971"}
Sep 09 23:56:27.878451 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.878433    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af57bfj"
Sep 09 23:56:27.878451 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:27.878460    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="341d497daba9fc73716d50703a4ecb46f12c70ec20d7fa0e37fa252e2b416971"
Sep 09 23:56:37.249621 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249593    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"]
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249886    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="pull"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249901    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="pull"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249915    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="util"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249922    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="util"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249944    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="extract"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.249953    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="extract"
Sep 09 23:56:37.250098 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.250005    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="310e75f9-ab8c-4c72-95d1-12022d833966" containerName="extract"
Sep 09 23:56:37.254322 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.254303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.256788 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.256762    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 23:56:37.256903 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.256792    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h5v4r\""
Sep 09 23:56:37.256903 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.256802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 23:56:37.264184 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.264165    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"]
Sep 09 23:56:37.402391 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.402358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6qw\" (UniqueName: \"kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.402507 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.402457    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.402591 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.402508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.503483 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.503415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.503483 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.503451    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.503483 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.503478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6qw\" (UniqueName: \"kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.503804 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.503786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.503838 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.503815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.515352 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.515328    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"]
Sep 09 23:56:37.518567 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.518533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.521229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.521208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 23:56:37.522784 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.522763    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 23:56:37.522884 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.522862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 23:56:37.522948 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.522885    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 23:56:37.522948 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.522939    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5zk9m\""
Sep 09 23:56:37.532845 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.532828    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6qw\" (UniqueName: \"kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.543715 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.543692    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"]
Sep 09 23:56:37.563249 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.563214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:37.604212 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.604180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.604355 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.604218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.604355 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.604292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jrh\" (UniqueName: \"kubernetes.io/projected/fd83a61d-b2c4-4ca6-afd2-090c5d875198-kube-api-access-89jrh\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.604355 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.604328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.604355 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.604350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.705533 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.705500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.705696 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.705560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.705696 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.705597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jrh\" (UniqueName: \"kubernetes.io/projected/fd83a61d-b2c4-4ca6-afd2-090c5d875198-kube-api-access-89jrh\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.705696 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.705617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.705696 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.705636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.706087 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.706067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.708106 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.708080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd83a61d-b2c4-4ca6-afd2-090c5d875198-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.708190 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.708109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.708267 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.708252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd83a61d-b2c4-4ca6-afd2-090c5d875198-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.715220 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.715200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jrh\" (UniqueName: \"kubernetes.io/projected/fd83a61d-b2c4-4ca6-afd2-090c5d875198-kube-api-access-89jrh\") pod \"opendatahub-operator-controller-manager-54cfd54796-ccgth\" (UID: \"fd83a61d-b2c4-4ca6-afd2-090c5d875198\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.828061 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.828031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:37.834718 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.834697    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"]
Sep 09 23:56:37.838851 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:37.838829    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59da049e_d6f4_4eda_b47f_1a39a0ece561.slice/crio-0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74 WatchSource:0}: Error finding container 0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74: Status 404 returned error can't find the container with id 0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74
Sep 09 23:56:37.923066 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.923026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerStarted","Data":"875da2dcd56ec8bcf264f8cb6d7063096ae1073d21d42d10085b286ac936e0c6"}
Sep 09 23:56:37.923181 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:37.923139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerStarted","Data":"0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74"}
Sep 09 23:56:38.107615 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:38.107513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"]
Sep 09 23:56:38.110091 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:56:38.110070    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd83a61d_b2c4_4ca6_afd2_090c5d875198.slice/crio-2b679d53cdab65a06bcd01f15204dd74c71d55d83ed69feb5b12d673bf7b4b53 WatchSource:0}: Error finding container 2b679d53cdab65a06bcd01f15204dd74c71d55d83ed69feb5b12d673bf7b4b53: Status 404 returned error can't find the container with id 2b679d53cdab65a06bcd01f15204dd74c71d55d83ed69feb5b12d673bf7b4b53
Sep 09 23:56:38.930693 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:38.930657    2576 generic.go:356] "Generic (PLEG): container finished" podID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerID="875da2dcd56ec8bcf264f8cb6d7063096ae1073d21d42d10085b286ac936e0c6" exitCode=0
Sep 09 23:56:38.931138 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:38.930739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerDied","Data":"875da2dcd56ec8bcf264f8cb6d7063096ae1073d21d42d10085b286ac936e0c6"}
Sep 09 23:56:38.933038 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:38.933004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth" event={"ID":"fd83a61d-b2c4-4ca6-afd2-090c5d875198","Type":"ContainerStarted","Data":"2b679d53cdab65a06bcd01f15204dd74c71d55d83ed69feb5b12d673bf7b4b53"}
Sep 09 23:56:41.948707 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:41.948678    2576 generic.go:356] "Generic (PLEG): container finished" podID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerID="9534f2bafb33aae2b8649a1106271d78e9f497e0e3b168c53c9d598260a7b731" exitCode=0
Sep 09 23:56:41.949120 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:41.948756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerDied","Data":"9534f2bafb33aae2b8649a1106271d78e9f497e0e3b168c53c9d598260a7b731"}
Sep 09 23:56:41.950383 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:41.950358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth" event={"ID":"fd83a61d-b2c4-4ca6-afd2-090c5d875198","Type":"ContainerStarted","Data":"7a4d01202b4f07e156e38a55f957c55be53dc37e2adc4285468a0e8eafbf18e0"}
Sep 09 23:56:42.956445 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:42.956411    2576 generic.go:356] "Generic (PLEG): container finished" podID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerID="bf89dbc04435eb42c99a44155977f8d6b97b79bd562ce0da84dfb94d20a9dd35" exitCode=0
Sep 09 23:56:42.956887 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:42.956478    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerDied","Data":"bf89dbc04435eb42c99a44155977f8d6b97b79bd562ce0da84dfb94d20a9dd35"}
Sep 09 23:56:42.958179 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:42.958158    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd83a61d-b2c4-4ca6-afd2-090c5d875198" containerID="7a4d01202b4f07e156e38a55f957c55be53dc37e2adc4285468a0e8eafbf18e0" exitCode=0
Sep 09 23:56:42.958299 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:42.958242    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth" event={"ID":"fd83a61d-b2c4-4ca6-afd2-090c5d875198","Type":"ContainerDied","Data":"7a4d01202b4f07e156e38a55f957c55be53dc37e2adc4285468a0e8eafbf18e0"}
Sep 09 23:56:43.964598 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:43.964564    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth" event={"ID":"fd83a61d-b2c4-4ca6-afd2-090c5d875198","Type":"ContainerStarted","Data":"cea6eb54136ba951d27347ec8a8577cf1d481c30e798ba5f9cf3b36e61bbba0e"}
Sep 09 23:56:43.990364 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:43.990309    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth" podStartSLOduration=2.54818819 podStartE2EDuration="6.990291074s" totalImagesPullingTime="3.415012407s" totalInitContainerRuntime="1.027090477s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:43.989028011 +0000 UTC m=+477.309404086" watchObservedRunningTime="2026-09-09 23:56:43.990291074 +0000 UTC m=+477.310667138"
Sep 09 23:56:44.090855 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.090832    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:44.260871 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.260796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle\") pod \"59da049e-d6f4-4eda-b47f-1a39a0ece561\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") "
Sep 09 23:56:44.261014 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.260897    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util\") pod \"59da049e-d6f4-4eda-b47f-1a39a0ece561\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") "
Sep 09 23:56:44.261014 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.260939    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx6qw\" (UniqueName: \"kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw\") pod \"59da049e-d6f4-4eda-b47f-1a39a0ece561\" (UID: \"59da049e-d6f4-4eda-b47f-1a39a0ece561\") "
Sep 09 23:56:44.261733 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.261703    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle" (OuterVolumeSpecName: "bundle") pod "59da049e-d6f4-4eda-b47f-1a39a0ece561" (UID: "59da049e-d6f4-4eda-b47f-1a39a0ece561"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:44.263153 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.263130    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw" (OuterVolumeSpecName: "kube-api-access-dx6qw") pod "59da049e-d6f4-4eda-b47f-1a39a0ece561" (UID: "59da049e-d6f4-4eda-b47f-1a39a0ece561"). InnerVolumeSpecName "kube-api-access-dx6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:56:44.265016 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.264997    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util" (OuterVolumeSpecName: "util") pod "59da049e-d6f4-4eda-b47f-1a39a0ece561" (UID: "59da049e-d6f4-4eda-b47f-1a39a0ece561"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:56:44.362200 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.362172    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx6qw\" (UniqueName: \"kubernetes.io/projected/59da049e-d6f4-4eda-b47f-1a39a0ece561-kube-api-access-dx6qw\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:44.362200 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.362195    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:44.362200 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.362205    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59da049e-d6f4-4eda-b47f-1a39a0ece561-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:56:44.969975 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.969947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg"
Sep 09 23:56:44.970348 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.969974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0r5glg" event={"ID":"59da049e-d6f4-4eda-b47f-1a39a0ece561","Type":"ContainerDied","Data":"0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74"}
Sep 09 23:56:44.970348 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.970016    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef4ce147d20d47fac8c4a09805bae0a1ea4825dae568e35d64484663187db74"
Sep 09 23:56:44.970348 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:44.970307    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:56:55.978739 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:56:55.978708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-ccgth"
Sep 09 23:57:09.253414 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253383    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bbd6h"]
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253649    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="extract"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253661    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="extract"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253679    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="pull"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253685    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="pull"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253692    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="util"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253697    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="util"
Sep 09 23:57:09.253830 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.253753    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59da049e-d6f4-4eda-b47f-1a39a0ece561" containerName="extract"
Sep 09 23:57:09.268106 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.268078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.269946 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.269627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bbd6h"]
Sep 09 23:57:09.270673 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.270654    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 23:57:09.339101 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.339066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmzv\" (UniqueName: \"kubernetes.io/projected/136ee0c6-0264-4cb8-be45-b97dfbec6894-kube-api-access-ljmzv\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.339287 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.339177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.439769 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.439732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.439960 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.439785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmzv\" (UniqueName: \"kubernetes.io/projected/136ee0c6-0264-4cb8-be45-b97dfbec6894-kube-api-access-ljmzv\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.439960 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:57:09.439889    2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 23:57:09.440059 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:57:09.439970    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs podName:136ee0c6-0264-4cb8-be45-b97dfbec6894 nodeName:}" failed. No retries permitted until 2026-09-09 23:57:09.939950076 +0000 UTC m=+503.260326123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs") pod "model-serving-api-54d89bd5fc-bbd6h" (UID: "136ee0c6-0264-4cb8-be45-b97dfbec6894") : secret "model-serving-api-tls" not found
Sep 09 23:57:09.448649 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.448618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmzv\" (UniqueName: \"kubernetes.io/projected/136ee0c6-0264-4cb8-be45-b97dfbec6894-kube-api-access-ljmzv\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.944353 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.944308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:09.947020 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:09.946989    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136ee0c6-0264-4cb8-be45-b97dfbec6894-tls-certs\") pod \"model-serving-api-54d89bd5fc-bbd6h\" (UID: \"136ee0c6-0264-4cb8-be45-b97dfbec6894\") " pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:10.180719 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:10.180684    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:10.455639 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:10.455613    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bbd6h"]
Sep 09 23:57:10.457535 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:57:10.457509    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136ee0c6_0264_4cb8_be45_b97dfbec6894.slice/crio-f8e2b4e815935fb61530ca8a916232839bea96f39b38c7e95b20aaf28a417140 WatchSource:0}: Error finding container f8e2b4e815935fb61530ca8a916232839bea96f39b38c7e95b20aaf28a417140: Status 404 returned error can't find the container with id f8e2b4e815935fb61530ca8a916232839bea96f39b38c7e95b20aaf28a417140
Sep 09 23:57:11.094813 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:11.094738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h" event={"ID":"136ee0c6-0264-4cb8-be45-b97dfbec6894","Type":"ContainerStarted","Data":"f8e2b4e815935fb61530ca8a916232839bea96f39b38c7e95b20aaf28a417140"}
Sep 09 23:57:13.106452 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.106416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h" event={"ID":"136ee0c6-0264-4cb8-be45-b97dfbec6894","Type":"ContainerStarted","Data":"5a4a826c14e4dc230e81a2f30c22829a4cfc79fd15d60bf1f20cfb61d0fba9d0"}
Sep 09 23:57:13.106883 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.106570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:13.122992 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.122941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h" podStartSLOduration=1.6972302080000001 podStartE2EDuration="4.122926718s" totalImagesPullingTime="2.42569651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:13.122059604 +0000 UTC m=+506.442435693" watchObservedRunningTime="2026-09-09 23:57:13.122926718 +0000 UTC m=+506.443302782"
Sep 09 23:57:13.974356 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.974319    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"]
Sep 09 23:57:13.977672 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.977650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:13.980196 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.980174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 23:57:13.981181 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.981161    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ggdqv\""
Sep 09 23:57:13.981288 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.981188    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 23:57:13.981441 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.981421    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 23:57:13.981526 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.981464    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 23:57:13.981606 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.981569    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:57:13.986921 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:13.986899    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"]
Sep 09 23:57:14.080413 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.080371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9ca9e1b-ebe6-4460-90b2-536f5317901e-manager-config\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.080629 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.080426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2jl\" (UniqueName: \"kubernetes.io/projected/d9ca9e1b-ebe6-4460-90b2-536f5317901e-kube-api-access-pv2jl\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.080629 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.080456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.080629 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.080480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.181038 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.181008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9ca9e1b-ebe6-4460-90b2-536f5317901e-manager-config\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.181485 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.181052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2jl\" (UniqueName: \"kubernetes.io/projected/d9ca9e1b-ebe6-4460-90b2-536f5317901e-kube-api-access-pv2jl\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.181485 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.181153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.181485 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.181215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.181813 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.181790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d9ca9e1b-ebe6-4460-90b2-536f5317901e-manager-config\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.183761 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.183740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.183950 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.183931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9ca9e1b-ebe6-4460-90b2-536f5317901e-cert\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.188764 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.188744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2jl\" (UniqueName: \"kubernetes.io/projected/d9ca9e1b-ebe6-4460-90b2-536f5317901e-kube-api-access-pv2jl\") pod \"lws-controller-manager-5895cdb77f-pn49w\" (UID: \"d9ca9e1b-ebe6-4460-90b2-536f5317901e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.288338 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.288256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:14.559185 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:14.559158    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"]
Sep 09 23:57:14.561504 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:57:14.561474    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ca9e1b_ebe6_4460_90b2_536f5317901e.slice/crio-48a366a38309d4a8dcd25532b29fedbc44fd73485835538da2797e71cbee91f5 WatchSource:0}: Error finding container 48a366a38309d4a8dcd25532b29fedbc44fd73485835538da2797e71cbee91f5: Status 404 returned error can't find the container with id 48a366a38309d4a8dcd25532b29fedbc44fd73485835538da2797e71cbee91f5
Sep 09 23:57:15.116262 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:15.116231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w" event={"ID":"d9ca9e1b-ebe6-4460-90b2-536f5317901e","Type":"ContainerStarted","Data":"48a366a38309d4a8dcd25532b29fedbc44fd73485835538da2797e71cbee91f5"}
Sep 09 23:57:17.126907 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:17.126874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w" event={"ID":"d9ca9e1b-ebe6-4460-90b2-536f5317901e","Type":"ContainerStarted","Data":"2889fd1667aa5a833e281d6f823a4c8f4008c047bd77a57db3968fe07b274703"}
Sep 09 23:57:17.127279 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:17.127021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:17.151137 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:17.151086    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w" podStartSLOduration=2.11404996 podStartE2EDuration="4.151072403s" totalImagesPullingTime="2.037022443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:17.149413938 +0000 UTC m=+510.469790003" watchObservedRunningTime="2026-09-09 23:57:17.151072403 +0000 UTC m=+510.471448468"
Sep 09 23:57:18.855341 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.855307    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"]
Sep 09 23:57:18.861636 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.861035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:18.863562 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.863512    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"]
Sep 09 23:57:18.864088 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.864065    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qrmbv\""
Sep 09 23:57:18.864651 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.864632    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 23:57:18.867185 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.867170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:18.869781 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.869764    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 23:57:18.870672 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.870651    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wljj5\""
Sep 09 23:57:18.885995 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.885974    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"]
Sep 09 23:57:18.887601 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.887579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"]
Sep 09 23:57:18.923340 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.923308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzll\" (UniqueName: \"kubernetes.io/projected/c253a960-4d9e-4366-8a1a-faec84e55cba-kube-api-access-rtzll\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:18.923505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.923355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxj2b\" (UniqueName: \"kubernetes.io/projected/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-kube-api-access-sxj2b\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:18.923505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.923435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:18.923613 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:18.923503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c253a960-4d9e-4366-8a1a-faec84e55cba-cert\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.024724 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.024688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxj2b\" (UniqueName: \"kubernetes.io/projected/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-kube-api-access-sxj2b\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:19.024919 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.024736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:19.024919 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.024766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c253a960-4d9e-4366-8a1a-faec84e55cba-cert\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.024919 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.024796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzll\" (UniqueName: \"kubernetes.io/projected/c253a960-4d9e-4366-8a1a-faec84e55cba-kube-api-access-rtzll\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.024919 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:57:19.024908    2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 23:57:19.025129 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:57:19.024976    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert podName:85b7d262-9bda-4a79-a77f-ccbf7c01d7e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:57:19.524956277 +0000 UTC m=+512.845332321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert") pod "llmisvc-controller-manager-777c6bd7d5-hcvtd" (UID: "85b7d262-9bda-4a79-a77f-ccbf7c01d7e1") : secret "llmisvc-webhook-server-cert" not found
Sep 09 23:57:19.027416 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.027397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c253a960-4d9e-4366-8a1a-faec84e55cba-cert\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.035148 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.035127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxj2b\" (UniqueName: \"kubernetes.io/projected/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-kube-api-access-sxj2b\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:19.035258 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.035164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzll\" (UniqueName: \"kubernetes.io/projected/c253a960-4d9e-4366-8a1a-faec84e55cba-kube-api-access-rtzll\") pod \"kserve-controller-manager-d6f4b65b8-76lnm\" (UID: \"c253a960-4d9e-4366-8a1a-faec84e55cba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.174442 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.174351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:19.469015 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.468943    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"]
Sep 09 23:57:19.472646 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:57:19.472620    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc253a960_4d9e_4366_8a1a_faec84e55cba.slice/crio-25387d50087cbaf03175c7ef59da7b0543c1b1dc44afcb1e93c2fe2330a605e0 WatchSource:0}: Error finding container 25387d50087cbaf03175c7ef59da7b0543c1b1dc44afcb1e93c2fe2330a605e0: Status 404 returned error can't find the container with id 25387d50087cbaf03175c7ef59da7b0543c1b1dc44afcb1e93c2fe2330a605e0
Sep 09 23:57:19.529267 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.529236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:19.531606 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.531590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b7d262-9bda-4a79-a77f-ccbf7c01d7e1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hcvtd\" (UID: \"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:19.780308 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:19.780226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:20.069042 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:20.068811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"]
Sep 09 23:57:20.071370 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:57:20.071342    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85b7d262_9bda_4a79_a77f_ccbf7c01d7e1.slice/crio-ef4995af56b1167b671ee54c9f1a5447f118de403ccb60e7c106fcb8c44c5a57 WatchSource:0}: Error finding container ef4995af56b1167b671ee54c9f1a5447f118de403ccb60e7c106fcb8c44c5a57: Status 404 returned error can't find the container with id ef4995af56b1167b671ee54c9f1a5447f118de403ccb60e7c106fcb8c44c5a57
Sep 09 23:57:20.143395 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:20.143358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd" event={"ID":"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1","Type":"ContainerStarted","Data":"ef4995af56b1167b671ee54c9f1a5447f118de403ccb60e7c106fcb8c44c5a57"}
Sep 09 23:57:20.145160 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:20.144989    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm" event={"ID":"c253a960-4d9e-4366-8a1a-faec84e55cba","Type":"ContainerStarted","Data":"25387d50087cbaf03175c7ef59da7b0543c1b1dc44afcb1e93c2fe2330a605e0"}
Sep 09 23:57:23.162568 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.162509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd" event={"ID":"85b7d262-9bda-4a79-a77f-ccbf7c01d7e1","Type":"ContainerStarted","Data":"0403b00b873405260f5376eda3d034cdd19012501cb580445df1ea8c6fcb2d76"}
Sep 09 23:57:23.162992 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.162600    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:23.164057 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.164038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm" event={"ID":"c253a960-4d9e-4366-8a1a-faec84e55cba","Type":"ContainerStarted","Data":"274780c8d320e5cb6c032de97d1dbe75f1b9e0256b08171fe450756622147c36"}
Sep 09 23:57:23.164179 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.164168    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:57:23.183023 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.182974    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd" podStartSLOduration=2.247370843 podStartE2EDuration="5.182960427s" totalImagesPullingTime="2.935589584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:23.182078769 +0000 UTC m=+516.502454835" watchObservedRunningTime="2026-09-09 23:57:23.182960427 +0000 UTC m=+516.503336491"
Sep 09 23:57:23.198317 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:23.198275    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm" podStartSLOduration=2.427877327 podStartE2EDuration="5.198257915s" totalImagesPullingTime="2.770380588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:23.197107539 +0000 UTC m=+516.517483604" watchObservedRunningTime="2026-09-09 23:57:23.198257915 +0000 UTC m=+516.518633980"
Sep 09 23:57:24.115354 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:24.115326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bbd6h"
Sep 09 23:57:28.134304 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:28.134272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-pn49w"
Sep 09 23:57:54.171660 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:54.171623    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hcvtd"
Sep 09 23:57:54.174657 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:57:54.174632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-76lnm"
Sep 09 23:58:27.932575 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.932531    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"]
Sep 09 23:58:27.936008 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.935989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:27.938409 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.938389    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 23:58:27.939216 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.939198    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 23:58:27.939325 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.939250    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h5v4r\""
Sep 09 23:58:27.943133 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.943110    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"]
Sep 09 23:58:27.955188 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.955166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:27.955294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.955209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:27.955294 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:27.955243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wpj\" (UniqueName: \"kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.034093 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.034055    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"]
Sep 09 23:58:28.037487 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.037463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.056366 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.056505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.056505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.056505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wpj\" (UniqueName: \"kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.056505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.056505 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqz2\" (UniqueName: \"kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.056810 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.056870 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.056830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.064463 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.064431    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wpj\" (UniqueName: \"kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.082424 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.082395    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"]
Sep 09 23:58:28.134405 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.134373    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"]
Sep 09 23:58:28.137793 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.137777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.145329 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.145294    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"]
Sep 09 23:58:28.156826 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.156924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t84h\" (UniqueName: \"kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.156924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.156924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.156924 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.157062 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.156932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqz2\" (UniqueName: \"kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.157229 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.157205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.157275 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.157215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.167716 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.167687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqz2\" (UniqueName: \"kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.245927 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.245848    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:28.258112 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.258081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.258228 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.258139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t84h\" (UniqueName: \"kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.258228 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.258191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.258472 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.258447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.258595 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.258473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.267888 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.267862    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"]
Sep 09 23:58:28.271979 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.271944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t84h\" (UniqueName: \"kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.272629 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.272610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.287774 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.287749    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"]
Sep 09 23:58:28.346630 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.346596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:28.358834 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.358807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.358957 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.358851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjpd\" (UniqueName: \"kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.358957 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.358927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.447202 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.447173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:28.459642 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.459619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.459724 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.459669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjpd\" (UniqueName: \"kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.459765 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.459733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.460006 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.459987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.460046 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.460003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.468880 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.468852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjpd\" (UniqueName: \"kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.522199 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.522174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"]
Sep 09 23:58:28.524358 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:58:28.524321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd626534_295a_498d_800f_ea6e3e4ffa29.slice/crio-7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e WatchSource:0}: Error finding container 7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e: Status 404 returned error can't find the container with id 7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e
Sep 09 23:58:28.588056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.588032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:28.620012 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.619904    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"]
Sep 09 23:58:28.644768 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:58:28.644726    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eb5de5_f71d_4f75_9ab5_b3612289d3ed.slice/crio-3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45 WatchSource:0}: Error finding container 3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45: Status 404 returned error can't find the container with id 3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45
Sep 09 23:58:28.716557 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.716513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"]
Sep 09 23:58:28.717949 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:58:28.717924    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13aaf33d_e0c1_41c2_b141_733a4911a24a.slice/crio-dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c WatchSource:0}: Error finding container dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c: Status 404 returned error can't find the container with id dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c
Sep 09 23:58:28.861773 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:28.861752    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"]
Sep 09 23:58:28.901358 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:58:28.901327    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8410918e_f28a_4271_8089_c7bea22a70e0.slice/crio-1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d WatchSource:0}: Error finding container 1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d: Status 404 returned error can't find the container with id 1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d
Sep 09 23:58:29.452809 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.452771    2576 generic.go:356] "Generic (PLEG): container finished" podID="8410918e-f28a-4271-8089-c7bea22a70e0" containerID="3429132ad474d1855e2d6cc920a871da1760c7a236b8f7ba62f9d11ea1df1f6d" exitCode=0
Sep 09 23:58:29.453265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.452856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4" event={"ID":"8410918e-f28a-4271-8089-c7bea22a70e0","Type":"ContainerDied","Data":"3429132ad474d1855e2d6cc920a871da1760c7a236b8f7ba62f9d11ea1df1f6d"}
Sep 09 23:58:29.453265 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.452906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4" event={"ID":"8410918e-f28a-4271-8089-c7bea22a70e0","Type":"ContainerStarted","Data":"1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d"}
Sep 09 23:58:29.454701 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.454674    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerID="d5bb2ead48da345f5227b6fab74ae931686737172d3f294838d5bdde30eb9cfc" exitCode=0
Sep 09 23:58:29.454820 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.454737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb" event={"ID":"fd626534-295a-498d-800f-ea6e3e4ffa29","Type":"ContainerDied","Data":"d5bb2ead48da345f5227b6fab74ae931686737172d3f294838d5bdde30eb9cfc"}
Sep 09 23:58:29.454820 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.454786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb" event={"ID":"fd626534-295a-498d-800f-ea6e3e4ffa29","Type":"ContainerStarted","Data":"7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e"}
Sep 09 23:58:29.456498 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.456474    2576 generic.go:356] "Generic (PLEG): container finished" podID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerID="0ffe556d144bf66d443271e205d93a1fd1be9c972bf91ebb8610126e6e37fd36" exitCode=0
Sep 09 23:58:29.456619 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.456500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h" event={"ID":"13aaf33d-e0c1-41c2-b141-733a4911a24a","Type":"ContainerDied","Data":"0ffe556d144bf66d443271e205d93a1fd1be9c972bf91ebb8610126e6e37fd36"}
Sep 09 23:58:29.456619 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.456567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h" event={"ID":"13aaf33d-e0c1-41c2-b141-733a4911a24a","Type":"ContainerStarted","Data":"dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c"}
Sep 09 23:58:29.458379 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.458356    2576 generic.go:356] "Generic (PLEG): container finished" podID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerID="84c27188ceb1d1b2a6513ebd4c3388407ace01a93847ea551e429e34ff7221fc" exitCode=0
Sep 09 23:58:29.458459 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.458393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t" event={"ID":"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed","Type":"ContainerDied","Data":"84c27188ceb1d1b2a6513ebd4c3388407ace01a93847ea551e429e34ff7221fc"}
Sep 09 23:58:29.458459 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:29.458423    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t" event={"ID":"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed","Type":"ContainerStarted","Data":"3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45"}
Sep 09 23:58:31.471240 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.471208    2576 generic.go:356] "Generic (PLEG): container finished" podID="8410918e-f28a-4271-8089-c7bea22a70e0" containerID="da21b7c79ecf96aa6a73e18265a0a32f3a3975996eb3103e6aebb179b5d198f9" exitCode=0
Sep 09 23:58:31.471692 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.471302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4" event={"ID":"8410918e-f28a-4271-8089-c7bea22a70e0","Type":"ContainerDied","Data":"da21b7c79ecf96aa6a73e18265a0a32f3a3975996eb3103e6aebb179b5d198f9"}
Sep 09 23:58:31.473302 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.473280    2576 generic.go:356] "Generic (PLEG): container finished" podID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerID="aaea649a06fdf7c82be2b4ad3d543ae8b37a676f97189c59077263fbb69940c6" exitCode=0
Sep 09 23:58:31.473403 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.473362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h" event={"ID":"13aaf33d-e0c1-41c2-b141-733a4911a24a","Type":"ContainerDied","Data":"aaea649a06fdf7c82be2b4ad3d543ae8b37a676f97189c59077263fbb69940c6"}
Sep 09 23:58:31.475471 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.475449    2576 generic.go:356] "Generic (PLEG): container finished" podID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerID="ffd19155aa013f1a1f58e48a7616dc767ee671ce5e7e2172a26af68b4dffadb8" exitCode=0
Sep 09 23:58:31.475588 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:31.475491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t" event={"ID":"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed","Type":"ContainerDied","Data":"ffd19155aa013f1a1f58e48a7616dc767ee671ce5e7e2172a26af68b4dffadb8"}
Sep 09 23:58:32.482414 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.482375    2576 generic.go:356] "Generic (PLEG): container finished" podID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerID="7735fb6ed190a3631af80ab2d34b4d22fbb1c6d4e514dc7f73795b61a20a9883" exitCode=0
Sep 09 23:58:32.482876 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.482466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t" event={"ID":"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed","Type":"ContainerDied","Data":"7735fb6ed190a3631af80ab2d34b4d22fbb1c6d4e514dc7f73795b61a20a9883"}
Sep 09 23:58:32.484633 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.484611    2576 generic.go:356] "Generic (PLEG): container finished" podID="8410918e-f28a-4271-8089-c7bea22a70e0" containerID="8ebbcb813656e82af00020bdbb972127a9dce56ab0fd79bd4116d9bf636835d6" exitCode=0
Sep 09 23:58:32.484770 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.484698    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4" event={"ID":"8410918e-f28a-4271-8089-c7bea22a70e0","Type":"ContainerDied","Data":"8ebbcb813656e82af00020bdbb972127a9dce56ab0fd79bd4116d9bf636835d6"}
Sep 09 23:58:32.486929 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.486910    2576 generic.go:356] "Generic (PLEG): container finished" podID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerID="7df72657ef715d72026030b0486b6b702f2ef398bf094d29f03536cc9183e5c4" exitCode=0
Sep 09 23:58:32.487017 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:32.486939    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h" event={"ID":"13aaf33d-e0c1-41c2-b141-733a4911a24a","Type":"ContainerDied","Data":"7df72657ef715d72026030b0486b6b702f2ef398bf094d29f03536cc9183e5c4"}
Sep 09 23:58:33.625515 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.625486    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:33.660978 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.660953    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:33.674525 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.674503    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:33.714176 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714140    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util\") pod \"13aaf33d-e0c1-41c2-b141-733a4911a24a\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") "
Sep 09 23:58:33.714176 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crjpd\" (UniqueName: \"kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd\") pod \"8410918e-f28a-4271-8089-c7bea22a70e0\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") "
Sep 09 23:58:33.714398 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714209    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle\") pod \"13aaf33d-e0c1-41c2-b141-733a4911a24a\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") "
Sep 09 23:58:33.714398 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714327    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util\") pod \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") "
Sep 09 23:58:33.714398 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714364    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t84h\" (UniqueName: \"kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h\") pod \"13aaf33d-e0c1-41c2-b141-733a4911a24a\" (UID: \"13aaf33d-e0c1-41c2-b141-733a4911a24a\") "
Sep 09 23:58:33.714398 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plqz2\" (UniqueName: \"kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2\") pod \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") "
Sep 09 23:58:33.714616 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714450    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util\") pod \"8410918e-f28a-4271-8089-c7bea22a70e0\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") "
Sep 09 23:58:33.714616 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714516    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle\") pod \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\" (UID: \"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed\") "
Sep 09 23:58:33.714616 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.714574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle\") pod \"8410918e-f28a-4271-8089-c7bea22a70e0\" (UID: \"8410918e-f28a-4271-8089-c7bea22a70e0\") "
Sep 09 23:58:33.715361 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.715309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle" (OuterVolumeSpecName: "bundle") pod "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" (UID: "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.715499 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.715460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle" (OuterVolumeSpecName: "bundle") pod "13aaf33d-e0c1-41c2-b141-733a4911a24a" (UID: "13aaf33d-e0c1-41c2-b141-733a4911a24a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.715679 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.715633    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle" (OuterVolumeSpecName: "bundle") pod "8410918e-f28a-4271-8089-c7bea22a70e0" (UID: "8410918e-f28a-4271-8089-c7bea22a70e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.715759 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.715717    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.715759 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.715737    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.716899 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.716880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd" (OuterVolumeSpecName: "kube-api-access-crjpd") pod "8410918e-f28a-4271-8089-c7bea22a70e0" (UID: "8410918e-f28a-4271-8089-c7bea22a70e0"). InnerVolumeSpecName "kube-api-access-crjpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:58:33.717367 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.717334    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h" (OuterVolumeSpecName: "kube-api-access-5t84h") pod "13aaf33d-e0c1-41c2-b141-733a4911a24a" (UID: "13aaf33d-e0c1-41c2-b141-733a4911a24a"). InnerVolumeSpecName "kube-api-access-5t84h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:58:33.718876 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.718855    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2" (OuterVolumeSpecName: "kube-api-access-plqz2") pod "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" (UID: "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed"). InnerVolumeSpecName "kube-api-access-plqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:58:33.720651 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.720620    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util" (OuterVolumeSpecName: "util") pod "13aaf33d-e0c1-41c2-b141-733a4911a24a" (UID: "13aaf33d-e0c1-41c2-b141-733a4911a24a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.722218 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.722197    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util" (OuterVolumeSpecName: "util") pod "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" (UID: "c1eb5de5-f71d-4f75-9ab5-b3612289d3ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.722750 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.722726    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util" (OuterVolumeSpecName: "util") pod "8410918e-f28a-4271-8089-c7bea22a70e0" (UID: "8410918e-f28a-4271-8089-c7bea22a70e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816858    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816903    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t84h\" (UniqueName: \"kubernetes.io/projected/13aaf33d-e0c1-41c2-b141-733a4911a24a-kube-api-access-5t84h\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816917    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqz2\" (UniqueName: \"kubernetes.io/projected/c1eb5de5-f71d-4f75-9ab5-b3612289d3ed-kube-api-access-plqz2\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816929    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816940    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8410918e-f28a-4271-8089-c7bea22a70e0-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816950    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13aaf33d-e0c1-41c2-b141-733a4911a24a-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:33.816961 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:33.816962    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjpd\" (UniqueName: \"kubernetes.io/projected/8410918e-f28a-4271-8089-c7bea22a70e0-kube-api-access-crjpd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:34.498727 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.498697    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h"
Sep 09 23:58:34.498908 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.498733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agc96h" event={"ID":"13aaf33d-e0c1-41c2-b141-733a4911a24a","Type":"ContainerDied","Data":"dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c"}
Sep 09 23:58:34.498908 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.498780    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd9c0f041d4b05d3d43e8f6e226914eb8d1c956dd4e3dc2219cecf445ee8d96c"
Sep 09 23:58:34.500801 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.500770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t" event={"ID":"c1eb5de5-f71d-4f75-9ab5-b3612289d3ed","Type":"ContainerDied","Data":"3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45"}
Sep 09 23:58:34.500959 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.500818    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd5c66e2b39fbac0afc31448c7832e3387f5977ed7dd20be6a7e5e0429a6c45"
Sep 09 23:58:34.500959 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.500782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74hq6t"
Sep 09 23:58:34.502970 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.502950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4" event={"ID":"8410918e-f28a-4271-8089-c7bea22a70e0","Type":"ContainerDied","Data":"1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d"}
Sep 09 23:58:34.503096 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.502980    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d91e7c84f84b0972bb4bcca11c8567fbc514aaa57c61738b349b884ff15565d"
Sep 09 23:58:34.503096 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:34.502989    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6dkcg4"
Sep 09 23:58:42.542617 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:42.542577    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerID="547a25b1d07234947b8d5c0f025b064608481758a112006a53e2f5800a455588" exitCode=0
Sep 09 23:58:42.543012 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:42.542661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb" event={"ID":"fd626534-295a-498d-800f-ea6e3e4ffa29","Type":"ContainerDied","Data":"547a25b1d07234947b8d5c0f025b064608481758a112006a53e2f5800a455588"}
Sep 09 23:58:43.550063 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:43.550025    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerID="6970fa646f750c1088c223db25aaef94d9aa626bd45556a713391e3c14316a4f" exitCode=0
Sep 09 23:58:43.550433 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:43.550085    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb" event={"ID":"fd626534-295a-498d-800f-ea6e3e4ffa29","Type":"ContainerDied","Data":"6970fa646f750c1088c223db25aaef94d9aa626bd45556a713391e3c14316a4f"}
Sep 09 23:58:44.676804 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.676782    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:44.819953 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.819926    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8wpj\" (UniqueName: \"kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj\") pod \"fd626534-295a-498d-800f-ea6e3e4ffa29\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") "
Sep 09 23:58:44.820116 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.820000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util\") pod \"fd626534-295a-498d-800f-ea6e3e4ffa29\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") "
Sep 09 23:58:44.820195 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.820154    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle\") pod \"fd626534-295a-498d-800f-ea6e3e4ffa29\" (UID: \"fd626534-295a-498d-800f-ea6e3e4ffa29\") "
Sep 09 23:58:44.820745 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.820644    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle" (OuterVolumeSpecName: "bundle") pod "fd626534-295a-498d-800f-ea6e3e4ffa29" (UID: "fd626534-295a-498d-800f-ea6e3e4ffa29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:44.822282 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.822257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj" (OuterVolumeSpecName: "kube-api-access-q8wpj") pod "fd626534-295a-498d-800f-ea6e3e4ffa29" (UID: "fd626534-295a-498d-800f-ea6e3e4ffa29"). InnerVolumeSpecName "kube-api-access-q8wpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:58:44.824582 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.824533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util" (OuterVolumeSpecName: "util") pod "fd626534-295a-498d-800f-ea6e3e4ffa29" (UID: "fd626534-295a-498d-800f-ea6e3e4ffa29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:58:44.921378 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.921328    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-bundle\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:44.921378 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.921370    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wpj\" (UniqueName: \"kubernetes.io/projected/fd626534-295a-498d-800f-ea6e3e4ffa29-kube-api-access-q8wpj\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:44.921378 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:44.921381    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd626534-295a-498d-800f-ea6e3e4ffa29-util\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 09 23:58:45.561688 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:45.561610    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb"
Sep 09 23:58:45.561843 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:45.561611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqj9fb" event={"ID":"fd626534-295a-498d-800f-ea6e3e4ffa29","Type":"ContainerDied","Data":"7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e"}
Sep 09 23:58:45.561843 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:58:45.561746    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2fa0e2454fe1101ffca4af34547d1b6bd622468a6055af0979284f1c79a28e"
Sep 09 23:59:11.383758 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.383677    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"]
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384013    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384026    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384033    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384038    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384045    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384054    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384066    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384074    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384087    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384095    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384104    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384112    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384122    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384128    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384134    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384142    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384160    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384165    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="extract"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384171    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384178    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384193    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384198    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="util"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384204    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="pull"
Sep 09 23:59:11.384225 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384208    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="pull"
Sep 09 23:59:11.385056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384277    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8410918e-f28a-4271-8089-c7bea22a70e0" containerName="extract"
Sep 09 23:59:11.385056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384288    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13aaf33d-e0c1-41c2-b141-733a4911a24a" containerName="extract"
Sep 09 23:59:11.385056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384294    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1eb5de5-f71d-4f75-9ab5-b3612289d3ed" containerName="extract"
Sep 09 23:59:11.385056 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.384301    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd626534-295a-498d-800f-ea6e3e4ffa29" containerName="extract"
Sep 09 23:59:11.387155 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.387139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.389575 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.389530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 23:59:11.389575 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.389569    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 23:59:11.389758 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.389641    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ptvt2\""
Sep 09 23:59:11.389758 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.389708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 23:59:11.390593 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.390577    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 23:59:11.394839 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.394820    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"]
Sep 09 23:59:11.426407 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.426372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k9p\" (UniqueName: \"kubernetes.io/projected/f2007dc4-5449-46d4-9227-7b87eeef29a9-kube-api-access-x4k9p\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.426515 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.426415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.426515 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.426464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2007dc4-5449-46d4-9227-7b87eeef29a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.526852 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.526824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.527015 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.526862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2007dc4-5449-46d4-9227-7b87eeef29a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.527015 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.526904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k9p\" (UniqueName: \"kubernetes.io/projected/f2007dc4-5449-46d4-9227-7b87eeef29a9-kube-api-access-x4k9p\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.527015 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:59:11.526994    2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 23:59:11.527141 ip-10-0-135-244 kubenswrapper[2576]: E0909 23:59:11.527072    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert podName:f2007dc4-5449-46d4-9227-7b87eeef29a9 nodeName:}" failed. No retries permitted until 2026-09-09 23:59:12.02705172 +0000 UTC m=+625.347427769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-tkqbd" (UID: "f2007dc4-5449-46d4-9227-7b87eeef29a9") : secret "plugin-serving-cert" not found
Sep 09 23:59:11.527742 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.527725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f2007dc4-5449-46d4-9227-7b87eeef29a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:11.535163 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:11.535137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k9p\" (UniqueName: \"kubernetes.io/projected/f2007dc4-5449-46d4-9227-7b87eeef29a9-kube-api-access-x4k9p\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:12.030880 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.030843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:12.033477 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.033443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2007dc4-5449-46d4-9227-7b87eeef29a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-tkqbd\" (UID: \"f2007dc4-5449-46d4-9227-7b87eeef29a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:12.297506 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.297427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"
Sep 09 23:59:12.570715 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.570691    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd"]
Sep 09 23:59:12.572728 ip-10-0-135-244 kubenswrapper[2576]: W0909 23:59:12.572703    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2007dc4_5449_46d4_9227_7b87eeef29a9.slice/crio-27894eb26d7b2a3336923cf33b78ac9c11d72fce5d9aaea4702aa7beba335aa0 WatchSource:0}: Error finding container 27894eb26d7b2a3336923cf33b78ac9c11d72fce5d9aaea4702aa7beba335aa0: Status 404 returned error can't find the container with id 27894eb26d7b2a3336923cf33b78ac9c11d72fce5d9aaea4702aa7beba335aa0
Sep 09 23:59:12.574117 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.574100    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:59:12.690906 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:12.690871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd" event={"ID":"f2007dc4-5449-46d4-9227-7b87eeef29a9","Type":"ContainerStarted","Data":"27894eb26d7b2a3336923cf33b78ac9c11d72fce5d9aaea4702aa7beba335aa0"}
Sep 09 23:59:17.727952 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:17.727916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd" event={"ID":"f2007dc4-5449-46d4-9227-7b87eeef29a9","Type":"ContainerStarted","Data":"0ee6478f8ab0bd90c28d4f8583da403420a3469a3191d875d0a6106b88c6c607"}
Sep 09 23:59:17.743179 ip-10-0-135-244 kubenswrapper[2576]: I0909 23:59:17.743128    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-tkqbd" podStartSLOduration=2.076950805 podStartE2EDuration="6.743113901s" totalImagesPullingTime="4.666163096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:59:17.742427107 +0000 UTC m=+631.062803172" watchObservedRunningTime="2026-09-09 23:59:17.743113901 +0000 UTC m=+631.063489960"
Sep 10 00:00:00.141779 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.141739    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29816640-m5z7h"]
Sep 10 00:00:00.162818 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.162790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29816640-m5z7h"]
Sep 10 00:00:00.162994 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.162895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.165615 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.165589    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Sep 10 00:00:00.165743 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.165669    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-fnmpt\""
Sep 10 00:00:00.331457 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.331419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.331646 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.331476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbf2\" (UniqueName: \"kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.433129 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.433042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbf2\" (UniqueName: \"kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.433267 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.433137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.433769 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.433752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.440967 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.440943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbf2\" (UniqueName: \"kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2\") pod \"image-pruner-29816640-m5z7h\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") " pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.486005 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.485979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:00.756723 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.756697    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29816640-m5z7h"]
Sep 10 00:00:00.758811 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:00:00.758782    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5333a2_7d15_4e8d_a087_45f5e2236aa8.slice/crio-b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9 WatchSource:0}: Error finding container b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9: Status 404 returned error can't find the container with id b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9
Sep 10 00:00:00.933820 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.933782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29816640-m5z7h" event={"ID":"fb5333a2-7d15-4e8d-a087-45f5e2236aa8","Type":"ContainerStarted","Data":"c6fea58aa97d184821f1cff17be76c2204ab24ad624a7f23c4f8d6cf136cd7e1"}
Sep 10 00:00:00.934023 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.933838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29816640-m5z7h" event={"ID":"fb5333a2-7d15-4e8d-a087-45f5e2236aa8","Type":"ContainerStarted","Data":"b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9"}
Sep 10 00:00:00.950457 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:00.950408    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29816640-m5z7h" podStartSLOduration=0.950393282 podStartE2EDuration="950.393282ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:00.948209588 +0000 UTC m=+674.268585652" watchObservedRunningTime="2026-09-10 00:00:00.950393282 +0000 UTC m=+674.270769345"
Sep 10 00:00:01.939935 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:01.939898    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb5333a2-7d15-4e8d-a087-45f5e2236aa8" containerID="c6fea58aa97d184821f1cff17be76c2204ab24ad624a7f23c4f8d6cf136cd7e1" exitCode=0
Sep 10 00:00:01.940317 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:01.939981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29816640-m5z7h" event={"ID":"fb5333a2-7d15-4e8d-a087-45f5e2236aa8","Type":"ContainerDied","Data":"c6fea58aa97d184821f1cff17be76c2204ab24ad624a7f23c4f8d6cf136cd7e1"}
Sep 10 00:00:03.083344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.083323    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:03.255903 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.255816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gbf2\" (UniqueName: \"kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2\") pod \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") "
Sep 10 00:00:03.256056 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.255917    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca\") pod \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\" (UID: \"fb5333a2-7d15-4e8d-a087-45f5e2236aa8\") "
Sep 10 00:00:03.256279 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.256259    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca" (OuterVolumeSpecName: "serviceca") pod "fb5333a2-7d15-4e8d-a087-45f5e2236aa8" (UID: "fb5333a2-7d15-4e8d-a087-45f5e2236aa8"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:00:03.258191 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.258166    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2" (OuterVolumeSpecName: "kube-api-access-8gbf2") pod "fb5333a2-7d15-4e8d-a087-45f5e2236aa8" (UID: "fb5333a2-7d15-4e8d-a087-45f5e2236aa8"). InnerVolumeSpecName "kube-api-access-8gbf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:03.356821 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.356777    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gbf2\" (UniqueName: \"kubernetes.io/projected/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-kube-api-access-8gbf2\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:00:03.356821 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.356813    2576 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5333a2-7d15-4e8d-a087-45f5e2236aa8-serviceca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:00:03.950931 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.950903    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29816640-m5z7h"
Sep 10 00:00:03.950931 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.950928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29816640-m5z7h" event={"ID":"fb5333a2-7d15-4e8d-a087-45f5e2236aa8","Type":"ContainerDied","Data":"b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9"}
Sep 10 00:00:03.951144 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:03.950970    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b175f6481ca5171c7dd065773bad8b3feba107bc0b6f5b97391ef3a77318c8c9"
Sep 10 00:00:05.783761 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.783723    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:05.784215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.784009    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5333a2-7d15-4e8d-a087-45f5e2236aa8" containerName="image-pruner"
Sep 10 00:00:05.784215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.784022    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5333a2-7d15-4e8d-a087-45f5e2236aa8" containerName="image-pruner"
Sep 10 00:00:05.784215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.784079    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5333a2-7d15-4e8d-a087-45f5e2236aa8" containerName="image-pruner"
Sep 10 00:00:05.787071 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.787054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:05.789410 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.789379    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 00:00:05.794167 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.793956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:05.878701 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.878669    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:05.978988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.978955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvzw\" (UniqueName: \"kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:05.979158 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:05.979019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.079557 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.079502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvzw\" (UniqueName: \"kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.079731 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.079561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.080148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.080131    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.086980 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.086952    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvzw\" (UniqueName: \"kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw\") pod \"limitador-limitador-8588749977-p5bm8\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") " pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.098811 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.098788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:06.368735 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.368660    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:06.371937 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:00:06.371910    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe9f5e3_43a2_41c1_9df3_646fc1e67760.slice/crio-bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a WatchSource:0}: Error finding container bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a: Status 404 returned error can't find the container with id bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a
Sep 10 00:00:06.965905 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:06.965870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" event={"ID":"efe9f5e3-43a2-41c1-9df3-646fc1e67760","Type":"ContainerStarted","Data":"bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a"}
Sep 10 00:00:18.021771 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:18.021675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" event={"ID":"efe9f5e3-43a2-41c1-9df3-646fc1e67760","Type":"ContainerStarted","Data":"53fee7a049b29295959d78cc05a35afba301e522f5fa44ac128a75333b548805"}
Sep 10 00:00:18.022146 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:18.021847    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:18.038250 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:18.038196    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" podStartSLOduration=1.710531854 podStartE2EDuration="13.038180769s" totalImagesPullingTime="11.327648915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:18.036225807 +0000 UTC m=+691.356601871" watchObservedRunningTime="2026-09-10 00:00:18.038180769 +0000 UTC m=+691.358556878"
Sep 10 00:00:26.137345 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:26.137301    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:26.137808 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:26.137564    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" podUID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" containerName="limitador" containerID="cri-o://53fee7a049b29295959d78cc05a35afba301e522f5fa44ac128a75333b548805" gracePeriod=30
Sep 10 00:00:26.138443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:26.138397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:27.069344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.069309    2576 generic.go:356] "Generic (PLEG): container finished" podID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" containerID="53fee7a049b29295959d78cc05a35afba301e522f5fa44ac128a75333b548805" exitCode=0
Sep 10 00:00:27.069489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.069379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" event={"ID":"efe9f5e3-43a2-41c1-9df3-646fc1e67760","Type":"ContainerDied","Data":"53fee7a049b29295959d78cc05a35afba301e522f5fa44ac128a75333b548805"}
Sep 10 00:00:27.069489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.069445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8" event={"ID":"efe9f5e3-43a2-41c1-9df3-646fc1e67760","Type":"ContainerDied","Data":"bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a"}
Sep 10 00:00:27.069489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.069468    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3ce1821c2440756e325438838f1aa3af9341fb4362e25d062c0dc45c81fd6a"
Sep 10 00:00:27.075319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.075302    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:27.154733 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.154708    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file\") pod \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") "
Sep 10 00:00:27.155070 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.154758    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smvzw\" (UniqueName: \"kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw\") pod \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\" (UID: \"efe9f5e3-43a2-41c1-9df3-646fc1e67760\") "
Sep 10 00:00:27.155070 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.155018    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file" (OuterVolumeSpecName: "config-file") pod "efe9f5e3-43a2-41c1-9df3-646fc1e67760" (UID: "efe9f5e3-43a2-41c1-9df3-646fc1e67760"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:00:27.157047 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.157025    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw" (OuterVolumeSpecName: "kube-api-access-smvzw") pod "efe9f5e3-43a2-41c1-9df3-646fc1e67760" (UID: "efe9f5e3-43a2-41c1-9df3-646fc1e67760"). InnerVolumeSpecName "kube-api-access-smvzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:27.255659 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.255612    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/efe9f5e3-43a2-41c1-9df3-646fc1e67760-config-file\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:00:27.255659 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:27.255651    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvzw\" (UniqueName: \"kubernetes.io/projected/efe9f5e3-43a2-41c1-9df3-646fc1e67760-kube-api-access-smvzw\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:00:28.074459 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:28.074431    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-p5bm8"
Sep 10 00:00:28.090886 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:28.090856    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:28.092603 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:28.092583    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-p5bm8"]
Sep 10 00:00:29.201873 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.201842    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" path="/var/lib/kubelet/pods/efe9f5e3-43a2-41c1-9df3-646fc1e67760/volumes"
Sep 10 00:00:29.924791 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.924754    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"]
Sep 10 00:00:29.925153 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.925136    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" containerName="limitador"
Sep 10 00:00:29.925233 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.925155    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" containerName="limitador"
Sep 10 00:00:29.925233 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.925220    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe9f5e3-43a2-41c1-9df3-646fc1e67760" containerName="limitador"
Sep 10 00:00:29.928517 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.928498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:29.931452 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.931425    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 00:00:29.931452 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.931445    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:00:29.932368 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.932347    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 00:00:29.932454 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.932349    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bkhnb\""
Sep 10 00:00:29.935691 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.935670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"]
Sep 10 00:00:29.980990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.980956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcfd\" (UniqueName: \"kubernetes.io/projected/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-kube-api-access-5kcfd\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:29.981180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:29.981018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-data\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.082001 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.081971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-data\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.082169 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.082048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcfd\" (UniqueName: \"kubernetes.io/projected/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-kube-api-access-5kcfd\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.082355 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.082339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-data\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.089855 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.089829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcfd\" (UniqueName: \"kubernetes.io/projected/75f19f7b-943f-4eb5-ba28-8bc244cd30c6-kube-api-access-5kcfd\") pod \"postgres-69fb4594cf-4rk2g\" (UID: \"75f19f7b-943f-4eb5-ba28-8bc244cd30c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.241009 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.240920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:30.513211 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:30.513182    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"]
Sep 10 00:00:30.514892 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:00:30.514865    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f19f7b_943f_4eb5_ba28_8bc244cd30c6.slice/crio-09c4e2140f4ab6a05e91daa6347f5165e24f48a57f5093b50b2230f90cd59dee WatchSource:0}: Error finding container 09c4e2140f4ab6a05e91daa6347f5165e24f48a57f5093b50b2230f90cd59dee: Status 404 returned error can't find the container with id 09c4e2140f4ab6a05e91daa6347f5165e24f48a57f5093b50b2230f90cd59dee
Sep 10 00:00:31.091183 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:31.091149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g" event={"ID":"75f19f7b-943f-4eb5-ba28-8bc244cd30c6","Type":"ContainerStarted","Data":"09c4e2140f4ab6a05e91daa6347f5165e24f48a57f5093b50b2230f90cd59dee"}
Sep 10 00:00:33.103466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:33.103384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g" event={"ID":"75f19f7b-943f-4eb5-ba28-8bc244cd30c6","Type":"ContainerStarted","Data":"606ec7878a9a880259cb02a67990d3bcaf48c731544aeeb258c26cbc28a110d2"}
Sep 10 00:00:33.103466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:33.103494    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:33.119040 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:33.118990    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g" podStartSLOduration=1.7486828829999999 podStartE2EDuration="4.118976421s" totalImagesPullingTime="2.370293538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:33.118316756 +0000 UTC m=+706.438692823" watchObservedRunningTime="2026-09-10 00:00:33.118976421 +0000 UTC m=+706.439352538"
Sep 10 00:00:39.137140 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.137059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4rk2g"
Sep 10 00:00:39.648285 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.648250    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-zlzqb"]
Sep 10 00:00:39.653006 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.652989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:39.655341 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.655319    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fwrps\""
Sep 10 00:00:39.658097 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.658073    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-zlzqb"]
Sep 10 00:00:39.711218 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.711185    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zlzqb"]
Sep 10 00:00:39.711424 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:00:39.711406    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tzrrd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-zlzqb" podUID="906838ae-6794-4f70-9003-363266cf5bf8"
Sep 10 00:00:39.759394 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.759358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrrd\" (UniqueName: \"kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd\") pod \"authorino-557f876f69-zlzqb\" (UID: \"906838ae-6794-4f70-9003-363266cf5bf8\") " pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:39.860320 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.860284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrrd\" (UniqueName: \"kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd\") pod \"authorino-557f876f69-zlzqb\" (UID: \"906838ae-6794-4f70-9003-363266cf5bf8\") " pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:39.868748 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:39.868722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrrd\" (UniqueName: \"kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd\") pod \"authorino-557f876f69-zlzqb\" (UID: \"906838ae-6794-4f70-9003-363266cf5bf8\") " pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:40.140133 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:40.140102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:40.147306 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:40.147284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:40.264058 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:40.264011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzrrd\" (UniqueName: \"kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd\") pod \"906838ae-6794-4f70-9003-363266cf5bf8\" (UID: \"906838ae-6794-4f70-9003-363266cf5bf8\") "
Sep 10 00:00:40.266349 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:40.266317    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd" (OuterVolumeSpecName: "kube-api-access-tzrrd") pod "906838ae-6794-4f70-9003-363266cf5bf8" (UID: "906838ae-6794-4f70-9003-363266cf5bf8"). InnerVolumeSpecName "kube-api-access-tzrrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:40.365592 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:40.365532    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrrd\" (UniqueName: \"kubernetes.io/projected/906838ae-6794-4f70-9003-363266cf5bf8-kube-api-access-tzrrd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:00:41.144424 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:41.144390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-zlzqb"
Sep 10 00:00:41.171495 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:41.171463    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-zlzqb"]
Sep 10 00:00:41.178056 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:41.178034    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-zlzqb"]
Sep 10 00:00:41.201620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:41.201594    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906838ae-6794-4f70-9003-363266cf5bf8" path="/var/lib/kubelet/pods/906838ae-6794-4f70-9003-363266cf5bf8/volumes"
Sep 10 00:00:45.786809 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.786775    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:00:45.793431 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.793407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:45.796297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.795992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 00:00:45.796297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.796048    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 00:00:45.796297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.796048    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cx96n\""
Sep 10 00:00:45.796297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.796051    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 00:00:45.800575 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.800524    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:00:45.909166 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.909132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:45.909166 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.909167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk2x\" (UniqueName: \"kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:45.909414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:45.909258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.010188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.010156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.010188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.010191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2x\" (UniqueName: \"kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.010465 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.010256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.010465 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:00:46.010332    2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 00:00:46.010465 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:00:46.010348    2576 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 00:00:46.010465 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:00:46.010417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert podName:b06371e8-ef52-438f-97a1-ab45767ca50c nodeName:}" failed. No retries permitted until 2026-09-10 00:00:46.510396592 +0000 UTC m=+719.830772653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert") pod "maas-controller-6fcf4d6694-kb2bg" (UID: "b06371e8-ef52-438f-97a1-ab45767ca50c") : secret "maas-controller-webhook-cert" not found
Sep 10 00:00:46.010465 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:00:46.010437    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs podName:b06371e8-ef52-438f-97a1-ab45767ca50c nodeName:}" failed. No retries permitted until 2026-09-10 00:00:46.51042848 +0000 UTC m=+719.830804529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs") pod "maas-controller-6fcf4d6694-kb2bg" (UID: "b06371e8-ef52-438f-97a1-ab45767ca50c") : secret "maas-controller-metrics-tls" not found
Sep 10 00:00:46.023270 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.023234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk2x\" (UniqueName: \"kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.516127 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.516093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.516127 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.516135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.518840 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.518815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.518986 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.518815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") pod \"maas-controller-6fcf4d6694-kb2bg\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") " pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.706947 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.706911    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:46.977910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:46.977879    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:00:46.979461 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:00:46.979437    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06371e8_ef52_438f_97a1_ab45767ca50c.slice/crio-c476f77e6742783f56e891d403056d3b7dee17646ecf065d8ac80be7f4875038 WatchSource:0}: Error finding container c476f77e6742783f56e891d403056d3b7dee17646ecf065d8ac80be7f4875038: Status 404 returned error can't find the container with id c476f77e6742783f56e891d403056d3b7dee17646ecf065d8ac80be7f4875038
Sep 10 00:00:47.175472 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:47.175435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" event={"ID":"b06371e8-ef52-438f-97a1-ab45767ca50c","Type":"ContainerStarted","Data":"c476f77e6742783f56e891d403056d3b7dee17646ecf065d8ac80be7f4875038"}
Sep 10 00:00:49.728976 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:49.728955    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 00:00:50.194158 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:50.194125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" event={"ID":"b06371e8-ef52-438f-97a1-ab45767ca50c","Type":"ContainerStarted","Data":"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"}
Sep 10 00:00:50.194408 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:50.194245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:00:50.213262 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:00:50.213211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" podStartSLOduration=2.468159362 podStartE2EDuration="5.213195058s" totalImagesPullingTime="2.745035696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:50.211519128 +0000 UTC m=+723.531895267" watchObservedRunningTime="2026-09-10 00:00:50.213195058 +0000 UTC m=+723.533571161"
Sep 10 00:01:01.201660 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.201632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:01:01.278670 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.278638    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:01:01.278942 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.278917    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" podUID="b06371e8-ef52-438f-97a1-ab45767ca50c" containerName="manager" containerID="cri-o://51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d" gracePeriod=30
Sep 10 00:01:01.525487 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.525464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:01:01.542832 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.542801    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") pod \"b06371e8-ef52-438f-97a1-ab45767ca50c\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") "
Sep 10 00:01:01.545260 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.545228    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b06371e8-ef52-438f-97a1-ab45767ca50c" (UID: "b06371e8-ef52-438f-97a1-ab45767ca50c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:01:01.643459 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.643421    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znk2x\" (UniqueName: \"kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x\") pod \"b06371e8-ef52-438f-97a1-ab45767ca50c\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") "
Sep 10 00:01:01.643459 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.643463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") pod \"b06371e8-ef52-438f-97a1-ab45767ca50c\" (UID: \"b06371e8-ef52-438f-97a1-ab45767ca50c\") "
Sep 10 00:01:01.643749 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.643660    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:01:01.645788 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.645754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b06371e8-ef52-438f-97a1-ab45767ca50c" (UID: "b06371e8-ef52-438f-97a1-ab45767ca50c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:01:01.645910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.645766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x" (OuterVolumeSpecName: "kube-api-access-znk2x") pod "b06371e8-ef52-438f-97a1-ab45767ca50c" (UID: "b06371e8-ef52-438f-97a1-ab45767ca50c"). InnerVolumeSpecName "kube-api-access-znk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:01:01.744022 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.743987    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znk2x\" (UniqueName: \"kubernetes.io/projected/b06371e8-ef52-438f-97a1-ab45767ca50c-kube-api-access-znk2x\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:01:01.744022 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:01.744014    2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b06371e8-ef52-438f-97a1-ab45767ca50c-webhook-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:01:02.250728 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.250698    2576 generic.go:356] "Generic (PLEG): container finished" podID="b06371e8-ef52-438f-97a1-ab45767ca50c" containerID="51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d" exitCode=0
Sep 10 00:01:02.251098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.250743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" event={"ID":"b06371e8-ef52-438f-97a1-ab45767ca50c","Type":"ContainerDied","Data":"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"}
Sep 10 00:01:02.251098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.250760    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg"
Sep 10 00:01:02.251098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.250806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fcf4d6694-kb2bg" event={"ID":"b06371e8-ef52-438f-97a1-ab45767ca50c","Type":"ContainerDied","Data":"c476f77e6742783f56e891d403056d3b7dee17646ecf065d8ac80be7f4875038"}
Sep 10 00:01:02.251098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.250838    2576 scope.go:117] "RemoveContainer" containerID="51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"
Sep 10 00:01:02.259857 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.259841    2576 scope.go:117] "RemoveContainer" containerID="51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"
Sep 10 00:01:02.260129 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:01:02.260110    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d\": container with ID starting with 51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d not found: ID does not exist" containerID="51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"
Sep 10 00:01:02.260174 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.260138    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d"} err="failed to get container status \"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d\": rpc error: code = NotFound desc = could not find container \"51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d\": container with ID starting with 51ac6633aee1eaf5c103f1a03ce7076cb356f1db491b7b64e61dda585229a28d not found: ID does not exist"
Sep 10 00:01:02.271460 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.271436    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:01:02.274915 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:02.274892    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fcf4d6694-kb2bg"]
Sep 10 00:01:03.201960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:03.201925    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b06371e8-ef52-438f-97a1-ab45767ca50c" path="/var/lib/kubelet/pods/b06371e8-ef52-438f-97a1-ab45767ca50c/volumes"
Sep 10 00:01:22.673293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.673257    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:01:22.673933 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.673727    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b06371e8-ef52-438f-97a1-ab45767ca50c" containerName="manager"
Sep 10 00:01:22.673933 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.673746    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b06371e8-ef52-438f-97a1-ab45767ca50c" containerName="manager"
Sep 10 00:01:22.673933 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.673855    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b06371e8-ef52-438f-97a1-ab45767ca50c" containerName="manager"
Sep 10 00:01:22.679392 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.679374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.681853 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.681834    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 00:01:22.681962 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.681836    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 00:01:22.688003 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.687500    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:01:22.704855 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.704827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.704983 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.704887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.704983 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.704947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.704983 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.704972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q45b\" (UniqueName: \"kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.805953 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.805911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.806157 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.806000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.806157 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.806028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q45b\" (UniqueName: \"kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.806157 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.806097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.806338 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:01:22.806154    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 10 00:01:22.806338 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:01:22.806208    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 10 00:01:22.806338 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:01:22.806230    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs podName:76c082a7-0baa-4df9-8431-007d7c0cf50f nodeName:}" failed. No retries permitted until 2026-09-10 00:01:23.306206886 +0000 UTC m=+756.626582930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs") pod "maas-api-75b96c447f-mcpkh" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f") : secret "maas-api-metrics-tls" not found
Sep 10 00:01:22.806338 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:01:22.806275    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls podName:76c082a7-0baa-4df9-8431-007d7c0cf50f nodeName:}" failed. No retries permitted until 2026-09-10 00:01:23.306255201 +0000 UTC m=+756.626631268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls") pod "maas-api-75b96c447f-mcpkh" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f") : secret "maas-api-serving-cert" not found
Sep 10 00:01:22.806685 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.806654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:22.816512 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:22.816489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q45b\" (UniqueName: \"kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.242861 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.242830    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7tq89"]
Sep 10 00:01:23.247645 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.247624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.250263 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.250231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 00:01:23.250263 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.250232    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 00:01:23.250443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.250231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 00:01:23.250443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.250341    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4j4d9\""
Sep 10 00:01:23.257349 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.257320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7tq89"]
Sep 10 00:01:23.310331 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.310303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.310466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.310363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxk9\" (UniqueName: \"kubernetes.io/projected/a07d06a7-aca4-4ae9-912f-5c5727565d09-kube-api-access-cxxk9\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.310466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.310412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.310466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.310443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07d06a7-aca4-4ae9-912f-5c5727565d09-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.312758 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.312735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.312874 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.312860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") pod \"maas-api-75b96c447f-mcpkh\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") " pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.411193 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.411158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07d06a7-aca4-4ae9-912f-5c5727565d09-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.411367 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.411233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxk9\" (UniqueName: \"kubernetes.io/projected/a07d06a7-aca4-4ae9-912f-5c5727565d09-kube-api-access-cxxk9\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.411976 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.411953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07d06a7-aca4-4ae9-912f-5c5727565d09-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.419123 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.419101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxk9\" (UniqueName: \"kubernetes.io/projected/a07d06a7-aca4-4ae9-912f-5c5727565d09-kube-api-access-cxxk9\") pod \"payload-processing-5b54b7fbb7-7tq89\" (UID: \"a07d06a7-aca4-4ae9-912f-5c5727565d09\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.559559 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.559454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:23.592908 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.592869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:23.849350 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.849315    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7tq89"]
Sep 10 00:01:23.851918 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:01:23.851894    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07d06a7_aca4_4ae9_912f_5c5727565d09.slice/crio-3ee37939a4e7881c4d6628276d669534cd8241b59b486e8b42ebf47fa902deae WatchSource:0}: Error finding container 3ee37939a4e7881c4d6628276d669534cd8241b59b486e8b42ebf47fa902deae: Status 404 returned error can't find the container with id 3ee37939a4e7881c4d6628276d669534cd8241b59b486e8b42ebf47fa902deae
Sep 10 00:01:23.878585 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:23.878532    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:01:23.881290 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:01:23.881269    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c082a7_0baa_4df9_8431_007d7c0cf50f.slice/crio-ed62708d80018138c7738c7a0983f48cc4a810ae07844c6f4a76dff298f4cf32 WatchSource:0}: Error finding container ed62708d80018138c7738c7a0983f48cc4a810ae07844c6f4a76dff298f4cf32: Status 404 returned error can't find the container with id ed62708d80018138c7738c7a0983f48cc4a810ae07844c6f4a76dff298f4cf32
Sep 10 00:01:24.367932 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:24.367899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" event={"ID":"76c082a7-0baa-4df9-8431-007d7c0cf50f","Type":"ContainerStarted","Data":"ed62708d80018138c7738c7a0983f48cc4a810ae07844c6f4a76dff298f4cf32"}
Sep 10 00:01:24.369265 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:24.369240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89" event={"ID":"a07d06a7-aca4-4ae9-912f-5c5727565d09","Type":"ContainerStarted","Data":"3ee37939a4e7881c4d6628276d669534cd8241b59b486e8b42ebf47fa902deae"}
Sep 10 00:01:26.382958 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.382918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89" event={"ID":"a07d06a7-aca4-4ae9-912f-5c5727565d09","Type":"ContainerStarted","Data":"321500a2f102703e02fb13eecf1eeeaec89071a73ba78ba883ea8dc06878fad0"}
Sep 10 00:01:26.383409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.382992    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:26.384622 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.384598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" event={"ID":"76c082a7-0baa-4df9-8431-007d7c0cf50f","Type":"ContainerStarted","Data":"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"}
Sep 10 00:01:26.384719 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.384654    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:26.398604 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.398555    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89" podStartSLOduration=1.044187618 podStartE2EDuration="3.398521653s" totalImagesPullingTime="2.354334035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:26.398053487 +0000 UTC m=+759.718429552" watchObservedRunningTime="2026-09-10 00:01:26.398521653 +0000 UTC m=+759.718897717"
Sep 10 00:01:26.415470 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:26.415415    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" podStartSLOduration=2.089968753 podStartE2EDuration="4.415399471s" totalImagesPullingTime="2.325430718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:26.412779094 +0000 UTC m=+759.733155159" watchObservedRunningTime="2026-09-10 00:01:26.415399471 +0000 UTC m=+759.735775575"
Sep 10 00:01:32.395420 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:32.395388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:01:37.391960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:37.391928    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7tq89"
Sep 10 00:01:39.061755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.061723    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"]
Sep 10 00:01:39.065205 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.065189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.067727 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.067704    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 00:01:39.067845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.067711    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 00:01:39.068468 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.068450    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pbjt6\""
Sep 10 00:01:39.068600 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.068476    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 00:01:39.072346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.072325    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"]
Sep 10 00:01:39.157772 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.157923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.157923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0339e05e-ad12-4251-baa3-6c611a21bb7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.157923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vs8w\" (UniqueName: \"kubernetes.io/projected/0339e05e-ad12-4251-baa3-6c611a21bb7d-kube-api-access-8vs8w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.157923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.158063 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.157939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258499 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258689 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258689 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258689 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0339e05e-ad12-4251-baa3-6c611a21bb7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vs8w\" (UniqueName: \"kubernetes.io/projected/0339e05e-ad12-4251-baa3-6c611a21bb7d-kube-api-access-8vs8w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258867 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258867 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.258963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.258947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.261226 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.261198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0339e05e-ad12-4251-baa3-6c611a21bb7d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.261343 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.261296    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0339e05e-ad12-4251-baa3-6c611a21bb7d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.266431 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.266414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vs8w\" (UniqueName: \"kubernetes.io/projected/0339e05e-ad12-4251-baa3-6c611a21bb7d-kube-api-access-8vs8w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps\" (UID: \"0339e05e-ad12-4251-baa3-6c611a21bb7d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.378007 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.377975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:39.652177 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:39.652106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"]
Sep 10 00:01:39.655060 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:01:39.655020    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0339e05e_ad12_4251_baa3_6c611a21bb7d.slice/crio-cc87131008d7d61d44f27fa08ced8c3cc440aa6d85ae06466de79b0ddb399c21 WatchSource:0}: Error finding container cc87131008d7d61d44f27fa08ced8c3cc440aa6d85ae06466de79b0ddb399c21: Status 404 returned error can't find the container with id cc87131008d7d61d44f27fa08ced8c3cc440aa6d85ae06466de79b0ddb399c21
Sep 10 00:01:40.463408 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:40.463368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps" event={"ID":"0339e05e-ad12-4251-baa3-6c611a21bb7d","Type":"ContainerStarted","Data":"cc87131008d7d61d44f27fa08ced8c3cc440aa6d85ae06466de79b0ddb399c21"}
Sep 10 00:01:41.470440 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:41.470401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps" event={"ID":"0339e05e-ad12-4251-baa3-6c611a21bb7d","Type":"ContainerStarted","Data":"29fc7324a984654c43928c9f8a7065dc454aa35abbca860f2390d0c41695e03c"}
Sep 10 00:01:41.488589 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:41.488526    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps" podStartSLOduration=1.320980006 podStartE2EDuration="2.488510025s" totalImagesPullingTime="1.167530019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:41.485774527 +0000 UTC m=+774.806150620" watchObservedRunningTime="2026-09-10 00:01:41.488510025 +0000 UTC m=+774.808886090"
Sep 10 00:01:49.378900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:49.378859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:49.379301 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:49.378922    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:49.391285 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:49.391260    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:01:49.525263 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:01:49.525237    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps"
Sep 10 00:02:10.861217 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.861141    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"]
Sep 10 00:02:10.865789 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.865770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.868076 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.868055    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 00:02:10.874246 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.874225    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"]
Sep 10 00:02:10.915985 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.915953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8968\" (UniqueName: \"kubernetes.io/projected/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-kube-api-access-v8968\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.916126 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.915989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.916126 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.916014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.916126 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.916079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.916251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.916134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:10.916251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:10.916156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016642 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.016842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.016812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8968\" (UniqueName: \"kubernetes.io/projected/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-kube-api-access-v8968\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.017507 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.017415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.017654 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.017510    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.017654 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.017459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.024274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.024251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.024429 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.024373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.026039 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.026018    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8968\" (UniqueName: \"kubernetes.io/projected/af1bf8c7-75d6-4d61-a71e-07ee85ea5b53-kube-api-access-v8968\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wbkkg\" (UID: \"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.176641 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.176554    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:11.447319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.447289    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"]
Sep 10 00:02:11.449939 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:02:11.449898    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1bf8c7_75d6_4d61_a71e_07ee85ea5b53.slice/crio-e974e1318a4aa9187a5f83cf7032cfff76c988618479beaf3c25373a63f28aa7 WatchSource:0}: Error finding container e974e1318a4aa9187a5f83cf7032cfff76c988618479beaf3c25373a63f28aa7: Status 404 returned error can't find the container with id e974e1318a4aa9187a5f83cf7032cfff76c988618479beaf3c25373a63f28aa7
Sep 10 00:02:11.627383 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:11.627355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg" event={"ID":"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53","Type":"ContainerStarted","Data":"e974e1318a4aa9187a5f83cf7032cfff76c988618479beaf3c25373a63f28aa7"}
Sep 10 00:02:12.633598 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:12.633558    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg" event={"ID":"af1bf8c7-75d6-4d61-a71e-07ee85ea5b53","Type":"ContainerStarted","Data":"7750c7574052baa21e86588619af629720c51e04362c8586ac616cf611a3cd7e"}
Sep 10 00:02:12.651277 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:12.651220    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg" podStartSLOduration=2.486991294 podStartE2EDuration="2.651201399s" totalImagesPullingTime="164.210105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:02:12.650595573 +0000 UTC m=+805.970971637" watchObservedRunningTime="2026-09-10 00:02:12.651201399 +0000 UTC m=+805.971577549"
Sep 10 00:02:21.177044 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:21.177007    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:21.177691 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:21.177086    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:21.189463 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:21.189434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:21.691119 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:21.691087    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wbkkg"
Sep 10 00:02:36.636489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.636459    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"]
Sep 10 00:02:36.641444 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.641426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.643911 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.643885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 00:02:36.650146 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.650128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"]
Sep 10 00:02:36.732062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5895\" (UniqueName: \"kubernetes.io/projected/2f49d8b6-1615-4958-8027-fb9816d04ca9-kube-api-access-d5895\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.732062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.732266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f49d8b6-1615-4958-8027-fb9816d04ca9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.732266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.732266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.732266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.732257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833076 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833076 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f49d8b6-1615-4958-8027-fb9816d04ca9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833479 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5895\" (UniqueName: \"kubernetes.io/projected/2f49d8b6-1615-4958-8027-fb9816d04ca9-kube-api-access-d5895\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833690 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.833739 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.833701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.835643 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.835622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f49d8b6-1615-4958-8027-fb9816d04ca9-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.835831 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.835811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f49d8b6-1615-4958-8027-fb9816d04ca9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.841969 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.841950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5895\" (UniqueName: \"kubernetes.io/projected/2f49d8b6-1615-4958-8027-fb9816d04ca9-kube-api-access-d5895\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp\" (UID: \"2f49d8b6-1615-4958-8027-fb9816d04ca9\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:36.952902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:36.952819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:37.223027 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:37.223000    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"]
Sep 10 00:02:37.224876 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:02:37.224852    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f49d8b6_1615_4958_8027_fb9816d04ca9.slice/crio-91f46ac5a29e33bcaf3c2995dd8739ff9656a242be87973e51095a95919b8ecf WatchSource:0}: Error finding container 91f46ac5a29e33bcaf3c2995dd8739ff9656a242be87973e51095a95919b8ecf: Status 404 returned error can't find the container with id 91f46ac5a29e33bcaf3c2995dd8739ff9656a242be87973e51095a95919b8ecf
Sep 10 00:02:37.764464 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:37.764422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp" event={"ID":"2f49d8b6-1615-4958-8027-fb9816d04ca9","Type":"ContainerStarted","Data":"97743631caf84d709aaca8508694e3cb4334c624ffb0b22c1a745145276b4936"}
Sep 10 00:02:37.764464 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:37.764467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp" event={"ID":"2f49d8b6-1615-4958-8027-fb9816d04ca9","Type":"ContainerStarted","Data":"91f46ac5a29e33bcaf3c2995dd8739ff9656a242be87973e51095a95919b8ecf"}
Sep 10 00:02:37.782851 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:37.782799    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp" podStartSLOduration=1.594678091 podStartE2EDuration="1.782783289s" totalImagesPullingTime="188.105198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:02:37.780454028 +0000 UTC m=+831.100830093" watchObservedRunningTime="2026-09-10 00:02:37.782783289 +0000 UTC m=+831.103159353"
Sep 10 00:02:46.953510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:46.953455    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:46.953937 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:46.953527    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:46.966283 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:46.966248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:02:47.826474 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:02:47.826442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp"
Sep 10 00:04:06.451792 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.451756    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:04:06.455063 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.455042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.458261 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.458244    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h47kt\""
Sep 10 00:04:06.458390 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.458352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f0f965-serving-cert\""
Sep 10 00:04:06.475556 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.475502    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:04:06.508107 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.508080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.508261 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.508113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.508261 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.508175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6x5\" (UniqueName: \"kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.508261 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.508201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.609512 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.609476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.609512 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.609512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.609794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.609562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6x5\" (UniqueName: \"kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.609794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.609582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.610261 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.610242    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.612389 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.612365    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.612503 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.612430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.617953 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.617927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6x5\" (UniqueName: \"kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5\") pod \"maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:06.765414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:06.765310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:07.046776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.046745    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:04:07.049090 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:07.049049    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5badaec_ba32_49fb_a75b_a42ea854358c.slice/crio-a487b64dee71fd145d2fef307a360d06f0a4e757cdd3533baa12a32e820da101 WatchSource:0}: Error finding container a487b64dee71fd145d2fef307a360d06f0a4e757cdd3533baa12a32e820da101: Status 404 returned error can't find the container with id a487b64dee71fd145d2fef307a360d06f0a4e757cdd3533baa12a32e820da101
Sep 10 00:04:07.213148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.213119    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:04:07.217836 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.217808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.220480 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.220455    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f0f965-dockercfg-khb6t\""
Sep 10 00:04:07.220646 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.220626    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f0f965\""
Sep 10 00:04:07.226482 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.226391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" event={"ID":"b5badaec-ba32-49fb-a75b-a42ea854358c","Type":"ContainerStarted","Data":"a487b64dee71fd145d2fef307a360d06f0a4e757cdd3533baa12a32e820da101"}
Sep 10 00:04:07.228591 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.228566    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:04:07.314158 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.314066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.314158 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.314136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29bc4\" (UniqueName: \"kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.415262 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.415230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.415418 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.415285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29bc4\" (UniqueName: \"kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.415938 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.415918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.424857 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.424833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29bc4\" (UniqueName: \"kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4\") pod \"payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.430267 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.430242    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:04:07.433789 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.433771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.436128 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.436111    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-01b7d5-serving-cert\""
Sep 10 00:04:07.444152 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.444129    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:04:07.516483 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.516465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.516831 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.516500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5p9\" (UniqueName: \"kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.516831 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.516616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.516831 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.516655    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.530065 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.530044    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:07.617808 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.617778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.617960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.617819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.617960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.617858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.617960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.617880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5p9\" (UniqueName: \"kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.618824 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.618797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.620888 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.620835    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.622346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.622327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.626963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.626938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5p9\" (UniqueName: \"kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9\") pod \"maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.748775 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.748746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:07.808628 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:07.808026    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:04:07.810757 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:07.810519    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063b8bec_0b47_42bf_827a_fa420de00dd8.slice/crio-8f210a14eb6925c90a41b9bf72446ec4e4f3cd16103560304f18af8db3c75309 WatchSource:0}: Error finding container 8f210a14eb6925c90a41b9bf72446ec4e4f3cd16103560304f18af8db3c75309: Status 404 returned error can't find the container with id 8f210a14eb6925c90a41b9bf72446ec4e4f3cd16103560304f18af8db3c75309
Sep 10 00:04:08.022924 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.022893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:04:08.024336 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:08.024304    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a714034_64a0_4635_a581_c0b2aa8fb150.slice/crio-34cf9b0f75976b8f7da7078248c6e766dab1571ee5b022371540da89d9c6e65d WatchSource:0}: Error finding container 34cf9b0f75976b8f7da7078248c6e766dab1571ee5b022371540da89d9c6e65d: Status 404 returned error can't find the container with id 34cf9b0f75976b8f7da7078248c6e766dab1571ee5b022371540da89d9c6e65d
Sep 10 00:04:08.234774 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.234745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" event={"ID":"b5badaec-ba32-49fb-a75b-a42ea854358c","Type":"ContainerStarted","Data":"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"}
Sep 10 00:04:08.234969 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.234820    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:08.236069 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.236043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" event={"ID":"0a714034-64a0-4635-a581-c0b2aa8fb150","Type":"ContainerStarted","Data":"34cf9b0f75976b8f7da7078248c6e766dab1571ee5b022371540da89d9c6e65d"}
Sep 10 00:04:08.237730 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.237705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" event={"ID":"063b8bec-0b47-42bf-827a-fa420de00dd8","Type":"ContainerStarted","Data":"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"}
Sep 10 00:04:08.237837 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.237754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" event={"ID":"063b8bec-0b47-42bf-827a-fa420de00dd8","Type":"ContainerStarted","Data":"8f210a14eb6925c90a41b9bf72446ec4e4f3cd16103560304f18af8db3c75309"}
Sep 10 00:04:08.237837 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.237785    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:08.259626 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.257030    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" podStartSLOduration=1.794890905 podStartE2EDuration="2.257011448s" totalImagesPullingTime="462.120543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:08.252495371 +0000 UTC m=+921.572871436" watchObservedRunningTime="2026-09-10 00:04:08.257011448 +0000 UTC m=+921.577387514"
Sep 10 00:04:08.273964 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.273904    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" podStartSLOduration=1.27388988 podStartE2EDuration="1.27388988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:08.272620177 +0000 UTC m=+921.592996255" watchObservedRunningTime="2026-09-10 00:04:08.27388988 +0000 UTC m=+921.594265943"
Sep 10 00:04:08.516222 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.516102    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:04:08.520371 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.520348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.522721 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.522697    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-cc0040-serving-cert\""
Sep 10 00:04:08.534494 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.534470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:04:08.627427 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.627394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.627600 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.627440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8krr\" (UniqueName: \"kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.627653 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.627607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.627653 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.627644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.728504 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.728461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.728656 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.728517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8krr\" (UniqueName: \"kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.728656 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:08.728638    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-serving-cert: secret "maas-api-e2e-worker-w4-cc0040-serving-cert" not found
Sep 10 00:04:08.728747 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.728659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.728747 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.728686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.728747 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:08.728704    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls podName:1037f845-0203-4257-80dc-5f0e8b084dc8 nodeName:}" failed. No retries permitted until 2026-09-10 00:04:09.228688744 +0000 UTC m=+922.549064789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls") pod "maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" (UID: "1037f845-0203-4257-80dc-5f0e8b084dc8") : secret "maas-api-e2e-worker-w4-cc0040-serving-cert" not found
Sep 10 00:04:08.729486 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.729405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.731303 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.731278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:08.740219 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:08.740198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8krr\" (UniqueName: \"kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:09.233274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:09.233244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:09.235799 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:09.235771    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") pod \"maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:09.438697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:09.438663    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:09.714799 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:09.714767    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:04:09.716055 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:09.716028    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1037f845_0203_4257_80dc_5f0e8b084dc8.slice/crio-7f9a326773fa3c4e4e88a7936504ff0d062deb5527ae4559e0bc5f38749829d5 WatchSource:0}: Error finding container 7f9a326773fa3c4e4e88a7936504ff0d062deb5527ae4559e0bc5f38749829d5: Status 404 returned error can't find the container with id 7f9a326773fa3c4e4e88a7936504ff0d062deb5527ae4559e0bc5f38749829d5
Sep 10 00:04:10.252906 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:10.252870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" event={"ID":"1037f845-0203-4257-80dc-5f0e8b084dc8","Type":"ContainerStarted","Data":"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"}
Sep 10 00:04:10.253088 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:10.252942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" event={"ID":"1037f845-0203-4257-80dc-5f0e8b084dc8","Type":"ContainerStarted","Data":"7f9a326773fa3c4e4e88a7936504ff0d062deb5527ae4559e0bc5f38749829d5"}
Sep 10 00:04:10.253088 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:10.252969    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:10.274825 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:10.274769    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" podStartSLOduration=1.920529452 podStartE2EDuration="2.274749198s" totalImagesPullingTime="354.219746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:10.274157233 +0000 UTC m=+923.594533298" watchObservedRunningTime="2026-09-10 00:04:10.274749198 +0000 UTC m=+923.595125263"
Sep 10 00:04:11.259893 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:11.259855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" event={"ID":"0a714034-64a0-4635-a581-c0b2aa8fb150","Type":"ContainerStarted","Data":"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"}
Sep 10 00:04:11.260299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:11.260091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:11.278651 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:11.278609    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" podStartSLOduration=1.911875673 podStartE2EDuration="4.27859663s" totalImagesPullingTime="2.366720957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:11.275737686 +0000 UTC m=+924.596113751" watchObservedRunningTime="2026-09-10 00:04:11.27859663 +0000 UTC m=+924.598972694"
Sep 10 00:04:13.893599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.893567    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:13.897385 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.897351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:13.900474 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.900452    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-46484ed0-serving-cert\""
Sep 10 00:04:13.916443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.916421    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:13.977192 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.976950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:13.977192 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.977013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822kz\" (UniqueName: \"kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:13.977192 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.977088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:13.977192 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:13.977110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.078596 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.078560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.078596 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.078600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.078845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.078686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.078845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.078713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822kz\" (UniqueName: \"kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.079407 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.079334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.081283 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.081263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.081349 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.081305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.087856 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.087833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822kz\" (UniqueName: \"kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz\") pod \"maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.216787 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.216699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:14.250938 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.250909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:04:14.493637 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.493613    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:14.495638 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:14.495572    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941b8252_bb60_4cff_903b_0a6c0f31ad06.slice/crio-e4db5edfd50b5f945f57e402fe248d4e01884408f748babcb13c1757dcb0ed77 WatchSource:0}: Error finding container e4db5edfd50b5f945f57e402fe248d4e01884408f748babcb13c1757dcb0ed77: Status 404 returned error can't find the container with id e4db5edfd50b5f945f57e402fe248d4e01884408f748babcb13c1757dcb0ed77
Sep 10 00:04:14.497118 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.497100    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:04:14.558622 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.558590    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:14.563137 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.563122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.565788 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.565762    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-46484ed0-dockercfg-d4pfs\""
Sep 10 00:04:14.565788 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.565783    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-46484ed0\""
Sep 10 00:04:14.573804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.573781    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:14.686985 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.686949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.687179 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.687003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppz5\" (UniqueName: \"kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.788039 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.787950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.788039 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.788017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppz5\" (UniqueName: \"kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.788693 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.788673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.795973 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.795955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppz5\" (UniqueName: \"kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5\") pod \"payload-processing-e2e-ait-46484ed0-777b7d677c-zl644\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") " pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:14.876043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:14.876026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:15.149973 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.149944    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:15.152220 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:15.152194    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b50763_2ef2_4376_9183_1e2b853c9936.slice/crio-88d28369f2ada68829f612cf33d90db14c2b4db69f2b0a0bc20ccf29bab4ef2a WatchSource:0}: Error finding container 88d28369f2ada68829f612cf33d90db14c2b4db69f2b0a0bc20ccf29bab4ef2a: Status 404 returned error can't find the container with id 88d28369f2ada68829f612cf33d90db14c2b4db69f2b0a0bc20ccf29bab4ef2a
Sep 10 00:04:15.295163 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.295117    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" event={"ID":"f9b50763-2ef2-4376-9183-1e2b853c9936","Type":"ContainerStarted","Data":"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"}
Sep 10 00:04:15.295360 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.295312    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:15.295508 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.295352    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" event={"ID":"f9b50763-2ef2-4376-9183-1e2b853c9936","Type":"ContainerStarted","Data":"88d28369f2ada68829f612cf33d90db14c2b4db69f2b0a0bc20ccf29bab4ef2a"}
Sep 10 00:04:15.296931 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.296900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" event={"ID":"941b8252-bb60-4cff-903b-0a6c0f31ad06","Type":"ContainerStarted","Data":"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"}
Sep 10 00:04:15.297026 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.296935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" event={"ID":"941b8252-bb60-4cff-903b-0a6c0f31ad06","Type":"ContainerStarted","Data":"e4db5edfd50b5f945f57e402fe248d4e01884408f748babcb13c1757dcb0ed77"}
Sep 10 00:04:15.297067 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.297026    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:15.312003 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.311950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" podStartSLOduration=1.311937858 podStartE2EDuration="1.311937858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:15.309847763 +0000 UTC m=+928.630223826" watchObservedRunningTime="2026-09-10 00:04:15.311937858 +0000 UTC m=+928.632313976"
Sep 10 00:04:15.327133 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:15.327075    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" podStartSLOduration=1.957123398 podStartE2EDuration="2.327060576s" totalImagesPullingTime="369.937178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:15.325876555 +0000 UTC m=+928.646252618" watchObservedRunningTime="2026-09-10 00:04:15.327060576 +0000 UTC m=+928.647436669"
Sep 10 00:04:16.265644 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:16.265613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:04:17.271529 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:17.271497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:04:18.304977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.304946    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:04:18.308889 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.308865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.311445 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.311420    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-39c7cb-serving-cert\""
Sep 10 00:04:18.324854 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.324826    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:04:18.422506 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.422465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.422506 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.422499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.422745 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.422527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.422745 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.422694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7gsl\" (UniqueName: \"kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.523868 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.523830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.523868 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.523867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.524083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.523906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.524083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.523990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7gsl\" (UniqueName: \"kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.524620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.524599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.526649 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.526625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.526746 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.526717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.532728 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.532708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7gsl\" (UniqueName: \"kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl\") pod \"maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.628795 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.628766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:18.906053 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:18.906028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:04:18.907637 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:18.907609    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c237d4d_3d5f_4a95_ba2d_d1d71928a3da.slice/crio-fb8a801b4f3b3458e9a3b72b73a3fe9ffebbc93c7f8f76e107bd134e9e10c677 WatchSource:0}: Error finding container fb8a801b4f3b3458e9a3b72b73a3fe9ffebbc93c7f8f76e107bd134e9e10c677: Status 404 returned error can't find the container with id fb8a801b4f3b3458e9a3b72b73a3fe9ffebbc93c7f8f76e107bd134e9e10c677
Sep 10 00:04:19.247286 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:19.247204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:04:19.332606 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:19.332573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" event={"ID":"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da","Type":"ContainerStarted","Data":"fb8a801b4f3b3458e9a3b72b73a3fe9ffebbc93c7f8f76e107bd134e9e10c677"}
Sep 10 00:04:21.310915 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:21.310883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:21.346358 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:21.346322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" event={"ID":"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da","Type":"ContainerStarted","Data":"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"}
Sep 10 00:04:21.346532 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:21.346438    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:21.362619 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:21.362562    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" podStartSLOduration=1.1313547800000001 podStartE2EDuration="3.362522426s" totalImagesPullingTime="2.231167646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:21.36165428 +0000 UTC m=+934.682030344" watchObservedRunningTime="2026-09-10 00:04:21.362522426 +0000 UTC m=+934.682898491"
Sep 10 00:04:26.307303 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:26.307268    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:27.357250 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:27.357219    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:04:36.092572 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.092400    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:04:36.097845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.097820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.100447 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.100242    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a6d968-serving-cert\""
Sep 10 00:04:36.111957 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.110702    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:04:36.291726 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.291683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.291934 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.291798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.291934 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.291859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.291934 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.291880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hgn\" (UniqueName: \"kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.393026 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.392945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.393174 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.393025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.393174 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.393055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hgn\" (UniqueName: \"kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.393174 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.393106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.393174 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:36.393161    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-serving-cert: secret "maas-api-e2e-inf-b-a6d968-serving-cert" not found
Sep 10 00:04:36.393338 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:36.393241    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls podName:86592b8b-5214-4600-8d08-8c95468489a3 nodeName:}" failed. No retries permitted until 2026-09-10 00:04:36.893218527 +0000 UTC m=+950.213594572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls") pod "maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" (UID: "86592b8b-5214-4600-8d08-8c95468489a3") : secret "maas-api-e2e-inf-b-a6d968-serving-cert" not found
Sep 10 00:04:36.393673 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.393654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.395781 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.395754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.401477 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.401453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hgn\" (UniqueName: \"kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.898717 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.898676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:36.901427 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:36.901395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") pod \"maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:37.014711 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:37.014673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:37.288026 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:37.287999    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:04:37.289332 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:37.289303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86592b8b_5214_4600_8d08_8c95468489a3.slice/crio-182e85dfce412ca6e8e2beab117ed01d8773dd246f2d6b9c3332070884b61751 WatchSource:0}: Error finding container 182e85dfce412ca6e8e2beab117ed01d8773dd246f2d6b9c3332070884b61751: Status 404 returned error can't find the container with id 182e85dfce412ca6e8e2beab117ed01d8773dd246f2d6b9c3332070884b61751
Sep 10 00:04:37.435903 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:37.435857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" event={"ID":"86592b8b-5214-4600-8d08-8c95468489a3","Type":"ContainerStarted","Data":"182e85dfce412ca6e8e2beab117ed01d8773dd246f2d6b9c3332070884b61751"}
Sep 10 00:04:38.443191 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:38.443154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" event={"ID":"86592b8b-5214-4600-8d08-8c95468489a3","Type":"ContainerStarted","Data":"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"}
Sep 10 00:04:38.443629 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:38.443291    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:38.461643 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:38.461593    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" podStartSLOduration=1.906519633 podStartE2EDuration="2.461579072s" totalImagesPullingTime="555.059439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:38.458100256 +0000 UTC m=+951.778476321" watchObservedRunningTime="2026-09-10 00:04:38.461579072 +0000 UTC m=+951.781955136"
Sep 10 00:04:44.455675 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:44.455646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:04:54.721807 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:54.721774    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:54.722377 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:54.722029    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" podUID="941b8252-bb60-4cff-903b-0a6c0f31ad06" containerName="maas-api" containerID="cri-o://bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54" gracePeriod=30
Sep 10 00:04:54.762622 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:54.762586    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:54.762880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:54.762854    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" podUID="f9b50763-2ef2-4376-9183-1e2b853c9936" containerName="payload-processing" containerID="cri-o://5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a" gracePeriod=30
Sep 10 00:04:55.036991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.036960    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:55.062737 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.062706    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:55.178435 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs\") pod \"941b8252-bb60-4cff-903b-0a6c0f31ad06\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") "
Sep 10 00:04:55.178632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-822kz\" (UniqueName: \"kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz\") pod \"941b8252-bb60-4cff-903b-0a6c0f31ad06\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") "
Sep 10 00:04:55.178632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nppz5\" (UniqueName: \"kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5\") pod \"f9b50763-2ef2-4376-9183-1e2b853c9936\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") "
Sep 10 00:04:55.178744 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls\") pod \"941b8252-bb60-4cff-903b-0a6c0f31ad06\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") "
Sep 10 00:04:55.178808 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca\") pod \"941b8252-bb60-4cff-903b-0a6c0f31ad06\" (UID: \"941b8252-bb60-4cff-903b-0a6c0f31ad06\") "
Sep 10 00:04:55.178808 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.178785    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume\") pod \"f9b50763-2ef2-4376-9183-1e2b853c9936\" (UID: \"f9b50763-2ef2-4376-9183-1e2b853c9936\") "
Sep 10 00:04:55.179234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.179198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "941b8252-bb60-4cff-903b-0a6c0f31ad06" (UID: "941b8252-bb60-4cff-903b-0a6c0f31ad06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:04:55.179515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.179257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9b50763-2ef2-4376-9183-1e2b853c9936" (UID: "f9b50763-2ef2-4376-9183-1e2b853c9936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:04:55.180909 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.180877    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz" (OuterVolumeSpecName: "kube-api-access-822kz") pod "941b8252-bb60-4cff-903b-0a6c0f31ad06" (UID: "941b8252-bb60-4cff-903b-0a6c0f31ad06"). InnerVolumeSpecName "kube-api-access-822kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:04:55.181004 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.180931    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "941b8252-bb60-4cff-903b-0a6c0f31ad06" (UID: "941b8252-bb60-4cff-903b-0a6c0f31ad06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:04:55.181159 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.181136    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "941b8252-bb60-4cff-903b-0a6c0f31ad06" (UID: "941b8252-bb60-4cff-903b-0a6c0f31ad06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:04:55.181213 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.181182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5" (OuterVolumeSpecName: "kube-api-access-nppz5") pod "f9b50763-2ef2-4376-9183-1e2b853c9936" (UID: "f9b50763-2ef2-4376-9183-1e2b853c9936"). InnerVolumeSpecName "kube-api-access-nppz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279490    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822kz\" (UniqueName: \"kubernetes.io/projected/941b8252-bb60-4cff-903b-0a6c0f31ad06-kube-api-access-822kz\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279526    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppz5\" (UniqueName: \"kubernetes.io/projected/f9b50763-2ef2-4376-9183-1e2b853c9936-kube-api-access-nppz5\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279572    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279587    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/941b8252-bb60-4cff-903b-0a6c0f31ad06-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279599    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9b50763-2ef2-4376-9183-1e2b853c9936-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.279620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.279611    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/941b8252-bb60-4cff-903b-0a6c0f31ad06-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.539696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.539598    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9b50763-2ef2-4376-9183-1e2b853c9936" containerID="5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a" exitCode=0
Sep 10 00:04:55.539696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.539651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"
Sep 10 00:04:55.539696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.539671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" event={"ID":"f9b50763-2ef2-4376-9183-1e2b853c9936","Type":"ContainerDied","Data":"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"}
Sep 10 00:04:55.539991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.539731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644" event={"ID":"f9b50763-2ef2-4376-9183-1e2b853c9936","Type":"ContainerDied","Data":"88d28369f2ada68829f612cf33d90db14c2b4db69f2b0a0bc20ccf29bab4ef2a"}
Sep 10 00:04:55.539991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.539764    2576 scope.go:117] "RemoveContainer" containerID="5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"
Sep 10 00:04:55.541135 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.541116    2576 generic.go:356] "Generic (PLEG): container finished" podID="941b8252-bb60-4cff-903b-0a6c0f31ad06" containerID="bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54" exitCode=0
Sep 10 00:04:55.541234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.541162    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"
Sep 10 00:04:55.541234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.541215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" event={"ID":"941b8252-bb60-4cff-903b-0a6c0f31ad06","Type":"ContainerDied","Data":"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"}
Sep 10 00:04:55.541340 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.541255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv" event={"ID":"941b8252-bb60-4cff-903b-0a6c0f31ad06","Type":"ContainerDied","Data":"e4db5edfd50b5f945f57e402fe248d4e01884408f748babcb13c1757dcb0ed77"}
Sep 10 00:04:55.549650 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.549631    2576 scope.go:117] "RemoveContainer" containerID="5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"
Sep 10 00:04:55.549922 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:55.549905    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a\": container with ID starting with 5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a not found: ID does not exist" containerID="5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"
Sep 10 00:04:55.549985 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.549928    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a"} err="failed to get container status \"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a\": rpc error: code = NotFound desc = could not find container \"5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a\": container with ID starting with 5616890fe2bb5ad784a890407fac3da77cd3ed8fb0a14ff6d95b717943651b9a not found: ID does not exist"
Sep 10 00:04:55.549985 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.549956    2576 scope.go:117] "RemoveContainer" containerID="bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"
Sep 10 00:04:55.558256 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.558239    2576 scope.go:117] "RemoveContainer" containerID="bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"
Sep 10 00:04:55.558517 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:04:55.558499    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54\": container with ID starting with bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54 not found: ID does not exist" containerID="bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"
Sep 10 00:04:55.558618 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.558522    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54"} err="failed to get container status \"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54\": rpc error: code = NotFound desc = could not find container \"bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54\": container with ID starting with bb48d60f419fc0c3b9717bb6333e05bc0760b6fb024016d657780015125efd54 not found: ID does not exist"
Sep 10 00:04:55.564910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.564890    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:55.568599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.568578    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46484ed0-777b7d677c-zl644"]
Sep 10 00:04:55.578786 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.578764    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:55.582718 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:55.582699    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46484ed0-54dc9f764f-s5hhv"]
Sep 10 00:04:57.207572 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:57.207526    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941b8252-bb60-4cff-903b-0a6c0f31ad06" path="/var/lib/kubelet/pods/941b8252-bb60-4cff-903b-0a6c0f31ad06/volumes"
Sep 10 00:04:57.207996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:57.207891    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b50763-2ef2-4376-9183-1e2b853c9936" path="/var/lib/kubelet/pods/f9b50763-2ef2-4376-9183-1e2b853c9936/volumes"
Sep 10 00:04:58.450822 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.450786    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:04:58.451402 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451381    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941b8252-bb60-4cff-903b-0a6c0f31ad06" containerName="maas-api"
Sep 10 00:04:58.451487 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451405    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="941b8252-bb60-4cff-903b-0a6c0f31ad06" containerName="maas-api"
Sep 10 00:04:58.451487 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451420    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b50763-2ef2-4376-9183-1e2b853c9936" containerName="payload-processing"
Sep 10 00:04:58.451487 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451428    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b50763-2ef2-4376-9183-1e2b853c9936" containerName="payload-processing"
Sep 10 00:04:58.451664 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451524    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b50763-2ef2-4376-9183-1e2b853c9936" containerName="payload-processing"
Sep 10 00:04:58.451664 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.451561    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="941b8252-bb60-4cff-903b-0a6c0f31ad06" containerName="maas-api"
Sep 10 00:04:58.456898 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.456264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.458648 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.458624    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e214cef3-serving-cert\""
Sep 10 00:04:58.463574 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.463515    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:04:58.615452 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.615405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.615452 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.615453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkfb\" (UniqueName: \"kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.615715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.615562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.615715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.615673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.716299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.716214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.716299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.716254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkfb\" (UniqueName: \"kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.716299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.716291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.716628 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.716460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.716982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.716955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.718888 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.718859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.718991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.718924    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.723801 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.723782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkfb\" (UniqueName: \"kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb\") pod \"maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:58.770249 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:58.770221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:59.025238 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.025153    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:04:59.029723 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.029705    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.032356 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.032333    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e214cef3\""
Sep 10 00:04:59.032504 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.032338    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e214cef3-dockercfg-vjljx\""
Sep 10 00:04:59.038353 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.038327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:04:59.052906 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.052822    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:04:59.057597 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:59.057565    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dae980_bd9e_41af_bbf7_9ac05d2da573.slice/crio-804892c6e57c593257987a2ccc35cbab63da1abbfa221bf3fead5f08bb92b818 WatchSource:0}: Error finding container 804892c6e57c593257987a2ccc35cbab63da1abbfa221bf3fead5f08bb92b818: Status 404 returned error can't find the container with id 804892c6e57c593257987a2ccc35cbab63da1abbfa221bf3fead5f08bb92b818
Sep 10 00:04:59.120502 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.120460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfj5g\" (UniqueName: \"kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.120707 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.120529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.221955 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.221923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfj5g\" (UniqueName: \"kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.222135 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.221974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.222670 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.222652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.229714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.229686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfj5g\" (UniqueName: \"kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g\") pod \"payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.342515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.342484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:04:59.566469 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.566428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" event={"ID":"c1dae980-bd9e-41af-bbf7-9ac05d2da573","Type":"ContainerStarted","Data":"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"}
Sep 10 00:04:59.566862 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.566489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" event={"ID":"c1dae980-bd9e-41af-bbf7-9ac05d2da573","Type":"ContainerStarted","Data":"804892c6e57c593257987a2ccc35cbab63da1abbfa221bf3fead5f08bb92b818"}
Sep 10 00:04:59.566862 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.566517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:04:59.583826 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.583784    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" podStartSLOduration=1.205931488 podStartE2EDuration="1.583768633s" totalImagesPullingTime="377.837145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:59.581980152 +0000 UTC m=+972.902356216" watchObservedRunningTime="2026-09-10 00:04:59.583768633 +0000 UTC m=+972.904144697"
Sep 10 00:04:59.612977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:04:59.612953    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:04:59.614198 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:04:59.614174    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b984f_9954_4dd5_8830_b7a3053de560.slice/crio-0a12919351262f3609281bf0d157cee4b54465e896c6c627f353119b5da543ae WatchSource:0}: Error finding container 0a12919351262f3609281bf0d157cee4b54465e896c6c627f353119b5da543ae: Status 404 returned error can't find the container with id 0a12919351262f3609281bf0d157cee4b54465e896c6c627f353119b5da543ae
Sep 10 00:05:00.574477 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:00.574433    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" event={"ID":"2a4b984f-9954-4dd5-8830-b7a3053de560","Type":"ContainerStarted","Data":"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"}
Sep 10 00:05:00.574996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:00.574603    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:05:00.574996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:00.574654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" event={"ID":"2a4b984f-9954-4dd5-8830-b7a3053de560","Type":"ContainerStarted","Data":"0a12919351262f3609281bf0d157cee4b54465e896c6c627f353119b5da543ae"}
Sep 10 00:05:00.590179 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:00.590118    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" podStartSLOduration=1.590101766 podStartE2EDuration="1.590101766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:00.589735615 +0000 UTC m=+973.910111692" watchObservedRunningTime="2026-09-10 00:05:00.590101766 +0000 UTC m=+973.910477832"
Sep 10 00:05:05.580771 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:05.580741    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:05:11.582076 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:11.581996    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:05:12.040464 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.040378    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:05:12.045374 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.045345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.048904 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.048880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"default-dockercfg-jt2wp\""
Sep 10 00:05:12.048904 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.048890    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"simulator-w4-cc0040-kserve-self-signed-certs\""
Sep 10 00:05:12.049086 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.048906    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"openshift-service-ca.crt\""
Sep 10 00:05:12.049086 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.048890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"kube-root-ca.crt\""
Sep 10 00:05:12.054154 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.054131    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:05:12.145613 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.145613 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.145872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.145872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.145872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.145990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.145900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptvc\" (UniqueName: \"kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.246880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.246847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247092 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.246947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptvc\" (UniqueName: \"kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247092 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.246975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247092 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.246997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247092 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.247075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247319 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.247096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247421 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.247398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247495 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.247445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.247568 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.247487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.249271 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.249246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.249770 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.249753    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.255045 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.255017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptvc\" (UniqueName: \"kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc\") pod \"simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") " pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.355647 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.355611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:12.633087 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.633060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:05:12.633721 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:05:12.633689    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd64469fe_af00_4922_a491_53156e92881a.slice/crio-997398e57698fe5d4a2e8efeddc53abf4ce7247d23131bc11884965d7df02fcd WatchSource:0}: Error finding container 997398e57698fe5d4a2e8efeddc53abf4ce7247d23131bc11884965d7df02fcd: Status 404 returned error can't find the container with id 997398e57698fe5d4a2e8efeddc53abf4ce7247d23131bc11884965d7df02fcd
Sep 10 00:05:12.643411 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:12.643391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" event={"ID":"d64469fe-af00-4922-a491-53156e92881a","Type":"ContainerStarted","Data":"997398e57698fe5d4a2e8efeddc53abf4ce7247d23131bc11884965d7df02fcd"}
Sep 10 00:05:13.650325 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:13.650286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" event={"ID":"d64469fe-af00-4922-a491-53156e92881a","Type":"ContainerStarted","Data":"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"}
Sep 10 00:05:13.667757 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:13.667708    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" podStartSLOduration=1.667692568 podStartE2EDuration="1.667692568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:13.667251935 +0000 UTC m=+986.987628000" watchObservedRunningTime="2026-09-10 00:05:13.667692568 +0000 UTC m=+986.988068632"
Sep 10 00:05:22.355765 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:22.355739    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:22.356135 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:22.355784    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:22.368186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:22.368164    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:22.712956 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:22.712878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:05:28.964623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:28.964586    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:05:28.965251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:28.964887    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" podUID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" containerName="maas-api" containerID="cri-o://f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc" gracePeriod=30
Sep 10 00:05:29.023557 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.023507    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:05:29.023791 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.023753    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" podUID="2a4b984f-9954-4dd5-8830-b7a3053de560" containerName="payload-pre-processing" containerID="cri-o://8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa" gracePeriod=30
Sep 10 00:05:29.271077 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.271051    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:05:29.308783 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.308691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls\") pod \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") "
Sep 10 00:05:29.308783 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.308755    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjkfb\" (UniqueName: \"kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb\") pod \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") "
Sep 10 00:05:29.308990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.308904    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs\") pod \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") "
Sep 10 00:05:29.308990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.308950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca\") pod \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\" (UID: \"c1dae980-bd9e-41af-bbf7-9ac05d2da573\") "
Sep 10 00:05:29.311734 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.311705    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1dae980-bd9e-41af-bbf7-9ac05d2da573" (UID: "c1dae980-bd9e-41af-bbf7-9ac05d2da573"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:05:29.311884 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.311763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1dae980-bd9e-41af-bbf7-9ac05d2da573" (UID: "c1dae980-bd9e-41af-bbf7-9ac05d2da573"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:05:29.311884 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.311867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1dae980-bd9e-41af-bbf7-9ac05d2da573" (UID: "c1dae980-bd9e-41af-bbf7-9ac05d2da573"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:05:29.312009 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.311966    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb" (OuterVolumeSpecName: "kube-api-access-cjkfb") pod "c1dae980-bd9e-41af-bbf7-9ac05d2da573" (UID: "c1dae980-bd9e-41af-bbf7-9ac05d2da573"). InnerVolumeSpecName "kube-api-access-cjkfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:05:29.317891 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.317868    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:05:29.410515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410480    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume\") pod \"2a4b984f-9954-4dd5-8830-b7a3053de560\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") "
Sep 10 00:05:29.410515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410518    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfj5g\" (UniqueName: \"kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g\") pod \"2a4b984f-9954-4dd5-8830-b7a3053de560\" (UID: \"2a4b984f-9954-4dd5-8830-b7a3053de560\") "
Sep 10 00:05:29.410740 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410694    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.410740 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410706    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1dae980-bd9e-41af-bbf7-9ac05d2da573-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.410740 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410717    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1dae980-bd9e-41af-bbf7-9ac05d2da573-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.410740 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410725    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjkfb\" (UniqueName: \"kubernetes.io/projected/c1dae980-bd9e-41af-bbf7-9ac05d2da573-kube-api-access-cjkfb\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.410925 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.410823    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a4b984f-9954-4dd5-8830-b7a3053de560" (UID: "2a4b984f-9954-4dd5-8830-b7a3053de560"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:05:29.412649 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.412629    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g" (OuterVolumeSpecName: "kube-api-access-jfj5g") pod "2a4b984f-9954-4dd5-8830-b7a3053de560" (UID: "2a4b984f-9954-4dd5-8830-b7a3053de560"). InnerVolumeSpecName "kube-api-access-jfj5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:05:29.512019 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.511937    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4b984f-9954-4dd5-8830-b7a3053de560-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.512019 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.511972    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfj5g\" (UniqueName: \"kubernetes.io/projected/2a4b984f-9954-4dd5-8830-b7a3053de560-kube-api-access-jfj5g\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.742927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.742892    2576 generic.go:356] "Generic (PLEG): container finished" podID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" containerID="f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc" exitCode=0
Sep 10 00:05:29.743121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.742938    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"
Sep 10 00:05:29.743121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.742981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" event={"ID":"c1dae980-bd9e-41af-bbf7-9ac05d2da573","Type":"ContainerDied","Data":"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"}
Sep 10 00:05:29.743121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.743034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx" event={"ID":"c1dae980-bd9e-41af-bbf7-9ac05d2da573","Type":"ContainerDied","Data":"804892c6e57c593257987a2ccc35cbab63da1abbfa221bf3fead5f08bb92b818"}
Sep 10 00:05:29.743121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.743056    2576 scope.go:117] "RemoveContainer" containerID="f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"
Sep 10 00:05:29.744455 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.744431    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a4b984f-9954-4dd5-8830-b7a3053de560" containerID="8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa" exitCode=0
Sep 10 00:05:29.744584 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.744474    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"
Sep 10 00:05:29.744584 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.744524    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" event={"ID":"2a4b984f-9954-4dd5-8830-b7a3053de560","Type":"ContainerDied","Data":"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"}
Sep 10 00:05:29.744696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.744594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp" event={"ID":"2a4b984f-9954-4dd5-8830-b7a3053de560","Type":"ContainerDied","Data":"0a12919351262f3609281bf0d157cee4b54465e896c6c627f353119b5da543ae"}
Sep 10 00:05:29.753811 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.753797    2576 scope.go:117] "RemoveContainer" containerID="f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"
Sep 10 00:05:29.754171 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:05:29.754106    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc\": container with ID starting with f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc not found: ID does not exist" containerID="f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"
Sep 10 00:05:29.754171 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.754132    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc"} err="failed to get container status \"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc\": rpc error: code = NotFound desc = could not find container \"f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc\": container with ID starting with f697ddb66989eaf2b1a8103d7e174ae1200aacd3f488b76e6a7c4836cd3175fc not found: ID does not exist"
Sep 10 00:05:29.754171 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.754148    2576 scope.go:117] "RemoveContainer" containerID="8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"
Sep 10 00:05:29.762699 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.762684    2576 scope.go:117] "RemoveContainer" containerID="8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"
Sep 10 00:05:29.762950 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:05:29.762934    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa\": container with ID starting with 8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa not found: ID does not exist" containerID="8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"
Sep 10 00:05:29.763012 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.762955    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa"} err="failed to get container status \"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa\": rpc error: code = NotFound desc = could not find container \"8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa\": container with ID starting with 8b0a8a1cc8c1a7a0584ab81ed5e417db461475fecbb5fa1a32d95719a5e140aa not found: ID does not exist"
Sep 10 00:05:29.770233 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.770208    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:05:29.779361 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.779343    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e214cef3-54f9ff7f96-h8pvx"]
Sep 10 00:05:29.788800 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.788779    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:05:29.792033 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:29.792014    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e214cef3-7ccdcd8c8b-lxkrp"]
Sep 10 00:05:31.203334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:31.203292    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4b984f-9954-4dd5-8830-b7a3053de560" path="/var/lib/kubelet/pods/2a4b984f-9954-4dd5-8830-b7a3053de560/volumes"
Sep 10 00:05:31.203751 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:31.203695    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" path="/var/lib/kubelet/pods/c1dae980-bd9e-41af-bbf7-9ac05d2da573/volumes"
Sep 10 00:05:33.422024 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.421980    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:05:33.422535 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422514    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" containerName="maas-api"
Sep 10 00:05:33.422535 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422556    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" containerName="maas-api"
Sep 10 00:05:33.422712 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422592    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4b984f-9954-4dd5-8830-b7a3053de560" containerName="payload-pre-processing"
Sep 10 00:05:33.422712 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422601    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4b984f-9954-4dd5-8830-b7a3053de560" containerName="payload-pre-processing"
Sep 10 00:05:33.422814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422713    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1dae980-bd9e-41af-bbf7-9ac05d2da573" containerName="maas-api"
Sep 10 00:05:33.422814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.422729    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4b984f-9954-4dd5-8830-b7a3053de560" containerName="payload-pre-processing"
Sep 10 00:05:33.427343 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.427324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.429752 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.429731    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2f357ac-serving-cert\""
Sep 10 00:05:33.436043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.436019    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:05:33.550593 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.550523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bt9t\" (UniqueName: \"kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.550803 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.550688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.550803 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.550750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.550803 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.550781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.651722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.651685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.651892 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.651727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.651892 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.651784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bt9t\" (UniqueName: \"kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.652004 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.651920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.652440 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.652415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.654303 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.654277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.654396 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.654322    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.659760 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.659738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bt9t\" (UniqueName: \"kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t\") pod \"maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:33.739186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:33.739118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:34.012474 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.012449    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:05:34.014747 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:05:34.014724    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e5e056_fd20_4164_a212_51836db0cea9.slice/crio-8eea5bfba01b2faf24e2bf01b581c5a8ee4b085c750cb02d0d80118f7af9c8fa WatchSource:0}: Error finding container 8eea5bfba01b2faf24e2bf01b581c5a8ee4b085c750cb02d0d80118f7af9c8fa: Status 404 returned error can't find the container with id 8eea5bfba01b2faf24e2bf01b581c5a8ee4b085c750cb02d0d80118f7af9c8fa
Sep 10 00:05:34.048930 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.048901    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:05:34.053173 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.053158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.055445 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.055423    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2f357ac-dockercfg-nfd4n\""
Sep 10 00:05:34.055587 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.055446    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2f357ac\""
Sep 10 00:05:34.063094 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.063072    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:05:34.156422 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.156387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.156737 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.156637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q9p\" (UniqueName: \"kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.257533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.257503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.257732 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.257618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q9p\" (UniqueName: \"kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.258229 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.258209    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.265317 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.265269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q9p\" (UniqueName: \"kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p\") pod \"payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.364430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.364390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.635902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.635869    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:05:34.638987 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:05:34.638960    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb23d31_0737_47bd_bcd3_c8700332556b.slice/crio-f9b3f38bf946bf3a43a98e313c0999d6143988308e77412e1dd8871a7936d970 WatchSource:0}: Error finding container f9b3f38bf946bf3a43a98e313c0999d6143988308e77412e1dd8871a7936d970: Status 404 returned error can't find the container with id f9b3f38bf946bf3a43a98e313c0999d6143988308e77412e1dd8871a7936d970
Sep 10 00:05:34.776111 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.776026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" event={"ID":"2fb23d31-0737-47bd-bcd3-c8700332556b","Type":"ContainerStarted","Data":"5bb91020f713719e97eda51444b9ebf3feff67652d561bbd85c4b819864c97d4"}
Sep 10 00:05:34.776111 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.776075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" event={"ID":"2fb23d31-0737-47bd-bcd3-c8700332556b","Type":"ContainerStarted","Data":"f9b3f38bf946bf3a43a98e313c0999d6143988308e77412e1dd8871a7936d970"}
Sep 10 00:05:34.776111 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.776096    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:34.777701 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.777677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" event={"ID":"93e5e056-fd20-4164-a212-51836db0cea9","Type":"ContainerStarted","Data":"a1c986a192693655d6d08793edf3825788def878ee930788dfc4a307ea091403"}
Sep 10 00:05:34.777837 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.777714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" event={"ID":"93e5e056-fd20-4164-a212-51836db0cea9","Type":"ContainerStarted","Data":"8eea5bfba01b2faf24e2bf01b581c5a8ee4b085c750cb02d0d80118f7af9c8fa"}
Sep 10 00:05:34.777837 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.777753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:34.792604 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.792561    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" podStartSLOduration=0.79253297 podStartE2EDuration="792.53297ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:34.790312347 +0000 UTC m=+1008.110688421" watchObservedRunningTime="2026-09-10 00:05:34.79253297 +0000 UTC m=+1008.112909035"
Sep 10 00:05:34.807379 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:34.807338    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" podStartSLOduration=1.345160938 podStartE2EDuration="1.807322409s" totalImagesPullingTime="462.161471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:34.806158161 +0000 UTC m=+1008.126534226" watchObservedRunningTime="2026-09-10 00:05:34.807322409 +0000 UTC m=+1008.127698473"
Sep 10 00:05:35.140023 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.139986    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:05:35.143364 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.143349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.145896 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.145874    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-39c7cb\"/\"kube-root-ca.crt\""
Sep 10 00:05:35.146195 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.146177    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-39c7cb\"/\"autores-model-39c7cb-kserve-self-signed-certs\""
Sep 10 00:05:35.146291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.146251    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-39c7cb\"/\"openshift-service-ca.crt\""
Sep 10 00:05:35.146892 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.146877    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-39c7cb\"/\"default-dockercfg-f9sqq\""
Sep 10 00:05:35.152692 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.152672    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:05:35.268667 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.268829 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.268829 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.268829 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268763    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.268829 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh947\" (UniqueName: \"kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.269070 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.268843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.369828 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.369797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.369839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.369885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.369904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.369937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh947\" (UniqueName: \"kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370345 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.370302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370434 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.370417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.370643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.370958 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.370673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.372648 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.372627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.372755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.372681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.377507 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.377484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh947\" (UniqueName: \"kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947\") pod \"autores-model-39c7cb-kserve-7b57d85cc5-d8dnp\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") " pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.453623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.453518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:35.735584 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.735554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:05:35.736647 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:05:35.736620    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d95224d_f019_409a_b326_e69020dfc6e8.slice/crio-9876436f688bea760f4323b60c252be28123f04648aefd2515f2947f6abeeb30 WatchSource:0}: Error finding container 9876436f688bea760f4323b60c252be28123f04648aefd2515f2947f6abeeb30: Status 404 returned error can't find the container with id 9876436f688bea760f4323b60c252be28123f04648aefd2515f2947f6abeeb30
Sep 10 00:05:35.784670 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:35.784635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" event={"ID":"2d95224d-f019-409a-b326-e69020dfc6e8","Type":"ContainerStarted","Data":"9876436f688bea760f4323b60c252be28123f04648aefd2515f2947f6abeeb30"}
Sep 10 00:05:36.791703 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:36.791670    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" event={"ID":"2d95224d-f019-409a-b326-e69020dfc6e8","Type":"ContainerStarted","Data":"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"}
Sep 10 00:05:36.808245 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:36.808194    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" podStartSLOduration=1.8081792970000001 podStartE2EDuration="1.808179297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:36.80628778 +0000 UTC m=+1010.126663845" watchObservedRunningTime="2026-09-10 00:05:36.808179297 +0000 UTC m=+1010.128555362"
Sep 10 00:05:40.798759 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:40.798732    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:05:45.453748 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:45.453713    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:45.454118 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:45.453766    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:45.467195 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:45.467170    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:05:45.787458 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:45.787379    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:05:45.857315 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:05:45.857288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:06:18.791318 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:18.791287    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:06:18.791840 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:18.791484    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" podUID="93e5e056-fd20-4164-a212-51836db0cea9" containerName="maas-api" containerID="cri-o://a1c986a192693655d6d08793edf3825788def878ee930788dfc4a307ea091403" gracePeriod=30
Sep 10 00:06:18.848021 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:18.847980    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:06:18.848236 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:18.848187    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" podUID="2fb23d31-0737-47bd-bcd3-c8700332556b" containerName="payload-pre-processing" containerID="cri-o://5bb91020f713719e97eda51444b9ebf3feff67652d561bbd85c4b819864c97d4" gracePeriod=30
Sep 10 00:06:19.031769 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.031665    2576 generic.go:356] "Generic (PLEG): container finished" podID="2fb23d31-0737-47bd-bcd3-c8700332556b" containerID="5bb91020f713719e97eda51444b9ebf3feff67652d561bbd85c4b819864c97d4" exitCode=0
Sep 10 00:06:19.031769 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.031728    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" event={"ID":"2fb23d31-0737-47bd-bcd3-c8700332556b","Type":"ContainerDied","Data":"5bb91020f713719e97eda51444b9ebf3feff67652d561bbd85c4b819864c97d4"}
Sep 10 00:06:19.033917 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.033855    2576 generic.go:356] "Generic (PLEG): container finished" podID="93e5e056-fd20-4164-a212-51836db0cea9" containerID="a1c986a192693655d6d08793edf3825788def878ee930788dfc4a307ea091403" exitCode=0
Sep 10 00:06:19.034043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.033914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" event={"ID":"93e5e056-fd20-4164-a212-51836db0cea9","Type":"ContainerDied","Data":"a1c986a192693655d6d08793edf3825788def878ee930788dfc4a307ea091403"}
Sep 10 00:06:19.096496 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.096468    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:06:19.130188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.130161    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:06:19.193363 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls\") pod \"93e5e056-fd20-4164-a212-51836db0cea9\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") "
Sep 10 00:06:19.193578 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193384    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs\") pod \"93e5e056-fd20-4164-a212-51836db0cea9\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") "
Sep 10 00:06:19.193578 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193408    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bt9t\" (UniqueName: \"kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t\") pod \"93e5e056-fd20-4164-a212-51836db0cea9\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") "
Sep 10 00:06:19.193578 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193474    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca\") pod \"93e5e056-fd20-4164-a212-51836db0cea9\" (UID: \"93e5e056-fd20-4164-a212-51836db0cea9\") "
Sep 10 00:06:19.193578 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4q9p\" (UniqueName: \"kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p\") pod \"2fb23d31-0737-47bd-bcd3-c8700332556b\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") "
Sep 10 00:06:19.193578 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193582    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume\") pod \"2fb23d31-0737-47bd-bcd3-c8700332556b\" (UID: \"2fb23d31-0737-47bd-bcd3-c8700332556b\") "
Sep 10 00:06:19.193977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.193945    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93e5e056-fd20-4164-a212-51836db0cea9" (UID: "93e5e056-fd20-4164-a212-51836db0cea9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:19.194048 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.194014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fb23d31-0737-47bd-bcd3-c8700332556b" (UID: "2fb23d31-0737-47bd-bcd3-c8700332556b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:19.195827 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.195800    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "93e5e056-fd20-4164-a212-51836db0cea9" (UID: "93e5e056-fd20-4164-a212-51836db0cea9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:06:19.195930 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.195848    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t" (OuterVolumeSpecName: "kube-api-access-6bt9t") pod "93e5e056-fd20-4164-a212-51836db0cea9" (UID: "93e5e056-fd20-4164-a212-51836db0cea9"). InnerVolumeSpecName "kube-api-access-6bt9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:19.196165 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.196137    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p" (OuterVolumeSpecName: "kube-api-access-r4q9p") pod "2fb23d31-0737-47bd-bcd3-c8700332556b" (UID: "2fb23d31-0737-47bd-bcd3-c8700332556b"). InnerVolumeSpecName "kube-api-access-r4q9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:19.196228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.196143    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93e5e056-fd20-4164-a212-51836db0cea9" (UID: "93e5e056-fd20-4164-a212-51836db0cea9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:06:19.295061 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295019    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e5e056-fd20-4164-a212-51836db0cea9-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.295061 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295051    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4q9p\" (UniqueName: \"kubernetes.io/projected/2fb23d31-0737-47bd-bcd3-c8700332556b-kube-api-access-r4q9p\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.295061 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295062    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb23d31-0737-47bd-bcd3-c8700332556b-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.295061 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295071    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.295340 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295080    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e5e056-fd20-4164-a212-51836db0cea9-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.295340 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:19.295087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bt9t\" (UniqueName: \"kubernetes.io/projected/93e5e056-fd20-4164-a212-51836db0cea9-kube-api-access-6bt9t\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.040768 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.040740    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"
Sep 10 00:06:20.040768 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.040746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt" event={"ID":"2fb23d31-0737-47bd-bcd3-c8700332556b","Type":"ContainerDied","Data":"f9b3f38bf946bf3a43a98e313c0999d6143988308e77412e1dd8871a7936d970"}
Sep 10 00:06:20.041346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.040815    2576 scope.go:117] "RemoveContainer" containerID="5bb91020f713719e97eda51444b9ebf3feff67652d561bbd85c4b819864c97d4"
Sep 10 00:06:20.042446 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.042415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml" event={"ID":"93e5e056-fd20-4164-a212-51836db0cea9","Type":"ContainerDied","Data":"8eea5bfba01b2faf24e2bf01b581c5a8ee4b085c750cb02d0d80118f7af9c8fa"}
Sep 10 00:06:20.042634 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.042600    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"
Sep 10 00:06:20.050830 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.050811    2576 scope.go:117] "RemoveContainer" containerID="a1c986a192693655d6d08793edf3825788def878ee930788dfc4a307ea091403"
Sep 10 00:06:20.061675 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.061616    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:06:20.065065 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.065044    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2f357ac-b66c7dd78-cz9wt"]
Sep 10 00:06:20.080200 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.080175    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:06:20.090299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.090275    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2f357ac-7c589ddbc4-dgkml"]
Sep 10 00:06:20.304610 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.304580    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:06:20.304852 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.304827    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" podUID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" containerName="maas-api" containerID="cri-o://6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64" gracePeriod=30
Sep 10 00:06:20.556770 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.556745    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:06:20.709856 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.709767    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls\") pod \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") "
Sep 10 00:06:20.710006 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.709885    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca\") pod \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") "
Sep 10 00:06:20.710006 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.709907    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7gsl\" (UniqueName: \"kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl\") pod \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") "
Sep 10 00:06:20.710006 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.709942    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs\") pod \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\" (UID: \"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da\") "
Sep 10 00:06:20.710320 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.710289    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" (UID: "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:20.712117 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.712079    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" (UID: "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:06:20.712298 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.712278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" (UID: "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:06:20.712351 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.712306    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl" (OuterVolumeSpecName: "kube-api-access-j7gsl") pod "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" (UID: "8c237d4d-3d5f-4a95-ba2d-d1d71928a3da"). InnerVolumeSpecName "kube-api-access-j7gsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:20.811149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.811116    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.811149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.811145    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7gsl\" (UniqueName: \"kubernetes.io/projected/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-kube-api-access-j7gsl\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.811149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.811154    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.811359 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:20.811163    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:21.049659 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.049568    2576 generic.go:356] "Generic (PLEG): container finished" podID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" containerID="6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64" exitCode=0
Sep 10 00:06:21.049659 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.049621    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"
Sep 10 00:06:21.050162 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.049660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" event={"ID":"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da","Type":"ContainerDied","Data":"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"}
Sep 10 00:06:21.050162 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.049766    2576 scope.go:117] "RemoveContainer" containerID="6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"
Sep 10 00:06:21.050274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.050156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr" event={"ID":"8c237d4d-3d5f-4a95-ba2d-d1d71928a3da","Type":"ContainerDied","Data":"fb8a801b4f3b3458e9a3b72b73a3fe9ffebbc93c7f8f76e107bd134e9e10c677"}
Sep 10 00:06:21.066658 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.066636    2576 scope.go:117] "RemoveContainer" containerID="6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"
Sep 10 00:06:21.066931 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:06:21.066910    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64\": container with ID starting with 6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64 not found: ID does not exist" containerID="6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"
Sep 10 00:06:21.066979 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.066941    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64"} err="failed to get container status \"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64\": rpc error: code = NotFound desc = could not find container \"6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64\": container with ID starting with 6c5846ba6f9c79c9060a7ff44b121e78e1b241011a2755962ed046c98b3aee64 not found: ID does not exist"
Sep 10 00:06:21.075234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.075206    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:06:21.078293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.078271    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-39c7cb-797d89c7db-dnwpr"]
Sep 10 00:06:21.202062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.202030    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb23d31-0737-47bd-bcd3-c8700332556b" path="/var/lib/kubelet/pods/2fb23d31-0737-47bd-bcd3-c8700332556b/volumes"
Sep 10 00:06:21.202519 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.202499    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" path="/var/lib/kubelet/pods/8c237d4d-3d5f-4a95-ba2d-d1d71928a3da/volumes"
Sep 10 00:06:21.202945 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:21.202926    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e5e056-fd20-4164-a212-51836db0cea9" path="/var/lib/kubelet/pods/93e5e056-fd20-4164-a212-51836db0cea9/volumes"
Sep 10 00:06:25.843039 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:25.842998    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:06:34.896162 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896129    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:06:34.896772 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" containerName="maas-api"
Sep 10 00:06:34.896850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896775    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" containerName="maas-api"
Sep 10 00:06:34.896850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896786    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e5e056-fd20-4164-a212-51836db0cea9" containerName="maas-api"
Sep 10 00:06:34.896850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896795    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e5e056-fd20-4164-a212-51836db0cea9" containerName="maas-api"
Sep 10 00:06:34.896850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896809    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb23d31-0737-47bd-bcd3-c8700332556b" containerName="payload-pre-processing"
Sep 10 00:06:34.896850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896817    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb23d31-0737-47bd-bcd3-c8700332556b" containerName="payload-pre-processing"
Sep 10 00:06:34.897105 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896928    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb23d31-0737-47bd-bcd3-c8700332556b" containerName="payload-pre-processing"
Sep 10 00:06:34.897105 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896944    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c237d4d-3d5f-4a95-ba2d-d1d71928a3da" containerName="maas-api"
Sep 10 00:06:34.897105 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.896974    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e5e056-fd20-4164-a212-51836db0cea9" containerName="maas-api"
Sep 10 00:06:34.901938 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.901913    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:34.904755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.904732    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-96c64491-serving-cert\""
Sep 10 00:06:34.911251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:34.911223    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:06:35.046499 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.046466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.046499 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.046510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgnp\" (UniqueName: \"kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.046773 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.046588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.046773 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.046759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.147346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.147249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.147346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.147300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.147608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.147417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgnp\" (UniqueName: \"kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.147608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.147474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.147725 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:06:35.147623    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-serving-cert: secret "maas-api-e2e-derive-96c64491-serving-cert" not found
Sep 10 00:06:35.147725 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:06:35.147703    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls podName:f15fd2a9-f566-4dfe-91ca-717d1795a314 nodeName:}" failed. No retries permitted until 2026-09-10 00:06:35.647684581 +0000 UTC m=+1068.968060623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls") pod "maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" (UID: "f15fd2a9-f566-4dfe-91ca-717d1795a314") : secret "maas-api-e2e-derive-96c64491-serving-cert" not found
Sep 10 00:06:35.148117 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.148093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.149996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.149974    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.155662 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.155641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgnp\" (UniqueName: \"kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.652972 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.652929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.655534 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.655514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") pod \"maas-api-e2e-derive-96c64491-ccbbdf749-qcwll\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:35.819131 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:35.819083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:36.097469 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:36.097430    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:06:36.098662 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:06:36.098629    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15fd2a9_f566_4dfe_91ca_717d1795a314.slice/crio-8428cf54fef75b52cf9f9556a4f8ebe0dd9fe37c82fb9adf2d1eead223cab9d3 WatchSource:0}: Error finding container 8428cf54fef75b52cf9f9556a4f8ebe0dd9fe37c82fb9adf2d1eead223cab9d3: Status 404 returned error can't find the container with id 8428cf54fef75b52cf9f9556a4f8ebe0dd9fe37c82fb9adf2d1eead223cab9d3
Sep 10 00:06:36.147810 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:36.147767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" event={"ID":"f15fd2a9-f566-4dfe-91ca-717d1795a314","Type":"ContainerStarted","Data":"8428cf54fef75b52cf9f9556a4f8ebe0dd9fe37c82fb9adf2d1eead223cab9d3"}
Sep 10 00:06:37.155941 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:37.155896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" event={"ID":"f15fd2a9-f566-4dfe-91ca-717d1795a314","Type":"ContainerStarted","Data":"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"}
Sep 10 00:06:37.156442 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:37.156038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:37.189188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:37.189132    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" podStartSLOduration=2.6833446480000003 podStartE2EDuration="3.189116834s" totalImagesPullingTime="505.772186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:06:37.186717429 +0000 UTC m=+1070.507093494" watchObservedRunningTime="2026-09-10 00:06:37.189116834 +0000 UTC m=+1070.509492897"
Sep 10 00:06:40.869718 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:40.869630    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" podUID="2d95224d-f019-409a-b326-e69020dfc6e8" containerName="main" containerID="cri-o://33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9" gracePeriod=45
Sep 10 00:06:41.127731 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.127339    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:06:41.178912 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.178882    2576 generic.go:356] "Generic (PLEG): container finished" podID="2d95224d-f019-409a-b326-e69020dfc6e8" containerID="33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9" exitCode=0
Sep 10 00:06:41.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.178927    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"
Sep 10 00:06:41.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.178972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" event={"ID":"2d95224d-f019-409a-b326-e69020dfc6e8","Type":"ContainerDied","Data":"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"}
Sep 10 00:06:41.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.179021    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp" event={"ID":"2d95224d-f019-409a-b326-e69020dfc6e8","Type":"ContainerDied","Data":"9876436f688bea760f4323b60c252be28123f04648aefd2515f2947f6abeeb30"}
Sep 10 00:06:41.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.179039    2576 scope.go:117] "RemoveContainer" containerID="33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"
Sep 10 00:06:41.187885 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.187856    2576 scope.go:117] "RemoveContainer" containerID="33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"
Sep 10 00:06:41.188132 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:06:41.188115    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9\": container with ID starting with 33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9 not found: ID does not exist" containerID="33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"
Sep 10 00:06:41.188230 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.188140    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9"} err="failed to get container status \"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9\": rpc error: code = NotFound desc = could not find container \"33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9\": container with ID starting with 33eca500d9bdbe22f8165bef31e94768f28fc5256caf2888b8fa8467a9795dc9 not found: ID does not exist"
Sep 10 00:06:41.218079 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218057    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218126    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh947\" (UniqueName: \"kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218210    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218360 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218281    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache\") pod \"2d95224d-f019-409a-b326-e69020dfc6e8\" (UID: \"2d95224d-f019-409a-b326-e69020dfc6e8\") "
Sep 10 00:06:41.218722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:06:41.218722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home" (OuterVolumeSpecName: "home") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:06:41.218722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218678    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.218722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218699    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.218722 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.218706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:06:41.220392 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.220370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm" (OuterVolumeSpecName: "dshm") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:06:41.220480 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.220405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:06:41.220521 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.220491    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947" (OuterVolumeSpecName: "kube-api-access-jh947") pod "2d95224d-f019-409a-b326-e69020dfc6e8" (UID: "2d95224d-f019-409a-b326-e69020dfc6e8"). InnerVolumeSpecName "kube-api-access-jh947". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:41.319759 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.319733    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.319759 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.319757    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d95224d-f019-409a-b326-e69020dfc6e8-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.319914 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.319769    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh947\" (UniqueName: \"kubernetes.io/projected/2d95224d-f019-409a-b326-e69020dfc6e8-kube-api-access-jh947\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.319914 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.319777    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d95224d-f019-409a-b326-e69020dfc6e8-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:06:41.501833 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.501804    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:06:41.503990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:41.503967    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-39c7cb/autores-model-39c7cb-kserve-7b57d85cc5-d8dnp"]
Sep 10 00:06:43.166461 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:43.166425    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:06:43.203643 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:43.203609    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d95224d-f019-409a-b326-e69020dfc6e8" path="/var/lib/kubelet/pods/2d95224d-f019-409a-b326-e69020dfc6e8/volumes"
Sep 10 00:06:47.229763 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:06:47.229733    2576 scope.go:117] "RemoveContainer" containerID="53fee7a049b29295959d78cc05a35afba301e522f5fa44ac128a75333b548805"
Sep 10 00:07:05.404199 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.404150    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:07:05.404711 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.404425    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" podUID="f15fd2a9-f566-4dfe-91ca-717d1795a314" containerName="maas-api" containerID="cri-o://71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1" gracePeriod=30
Sep 10 00:07:05.647814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.647790    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:07:05.753292 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.753208    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swgnp\" (UniqueName: \"kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp\") pod \"f15fd2a9-f566-4dfe-91ca-717d1795a314\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") "
Sep 10 00:07:05.753292 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.753256    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs\") pod \"f15fd2a9-f566-4dfe-91ca-717d1795a314\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") "
Sep 10 00:07:05.753489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.753301    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca\") pod \"f15fd2a9-f566-4dfe-91ca-717d1795a314\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") "
Sep 10 00:07:05.753489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.753326    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") pod \"f15fd2a9-f566-4dfe-91ca-717d1795a314\" (UID: \"f15fd2a9-f566-4dfe-91ca-717d1795a314\") "
Sep 10 00:07:05.753710 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.753679    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f15fd2a9-f566-4dfe-91ca-717d1795a314" (UID: "f15fd2a9-f566-4dfe-91ca-717d1795a314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:07:05.755635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.755611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp" (OuterVolumeSpecName: "kube-api-access-swgnp") pod "f15fd2a9-f566-4dfe-91ca-717d1795a314" (UID: "f15fd2a9-f566-4dfe-91ca-717d1795a314"). InnerVolumeSpecName "kube-api-access-swgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:07:05.755635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.755618    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f15fd2a9-f566-4dfe-91ca-717d1795a314" (UID: "f15fd2a9-f566-4dfe-91ca-717d1795a314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:07:05.755751 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.755637    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f15fd2a9-f566-4dfe-91ca-717d1795a314" (UID: "f15fd2a9-f566-4dfe-91ca-717d1795a314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:07:05.854432 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.854392    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgnp\" (UniqueName: \"kubernetes.io/projected/f15fd2a9-f566-4dfe-91ca-717d1795a314-kube-api-access-swgnp\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.854432 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.854425    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.854432 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.854435    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15fd2a9-f566-4dfe-91ca-717d1795a314-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.854432 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:05.854443    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f15fd2a9-f566-4dfe-91ca-717d1795a314-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:07:06.326259 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.326220    2576 generic.go:356] "Generic (PLEG): container finished" podID="f15fd2a9-f566-4dfe-91ca-717d1795a314" containerID="71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1" exitCode=0
Sep 10 00:07:06.326443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.326268    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"
Sep 10 00:07:06.326443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.326309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" event={"ID":"f15fd2a9-f566-4dfe-91ca-717d1795a314","Type":"ContainerDied","Data":"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"}
Sep 10 00:07:06.326443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.326365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll" event={"ID":"f15fd2a9-f566-4dfe-91ca-717d1795a314","Type":"ContainerDied","Data":"8428cf54fef75b52cf9f9556a4f8ebe0dd9fe37c82fb9adf2d1eead223cab9d3"}
Sep 10 00:07:06.326443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.326390    2576 scope.go:117] "RemoveContainer" containerID="71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"
Sep 10 00:07:06.336935 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.336914    2576 scope.go:117] "RemoveContainer" containerID="71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"
Sep 10 00:07:06.337269 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:07:06.337244    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1\": container with ID starting with 71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1 not found: ID does not exist" containerID="71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"
Sep 10 00:07:06.337378 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.337281    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1"} err="failed to get container status \"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1\": rpc error: code = NotFound desc = could not find container \"71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1\": container with ID starting with 71276d34a9fd5c1d56b14f248d759222d46aa92fcd7ffdcbfd7e3b695d63dfb1 not found: ID does not exist"
Sep 10 00:07:06.356697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.356656    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:07:06.362138 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:06.362110    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-96c64491-ccbbdf749-qcwll"]
Sep 10 00:07:07.209877 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.209841    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f15fd2a9-f566-4dfe-91ca-717d1795a314" path="/var/lib/kubelet/pods/f15fd2a9-f566-4dfe-91ca-717d1795a314/volumes"
Sep 10 00:07:07.528059 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.527973    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:07:07.528461 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528442    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15fd2a9-f566-4dfe-91ca-717d1795a314" containerName="maas-api"
Sep 10 00:07:07.528597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528463    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15fd2a9-f566-4dfe-91ca-717d1795a314" containerName="maas-api"
Sep 10 00:07:07.528597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528482    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d95224d-f019-409a-b326-e69020dfc6e8" containerName="main"
Sep 10 00:07:07.528597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528489    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d95224d-f019-409a-b326-e69020dfc6e8" containerName="main"
Sep 10 00:07:07.528597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528596    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15fd2a9-f566-4dfe-91ca-717d1795a314" containerName="maas-api"
Sep 10 00:07:07.528809 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.528613    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d95224d-f019-409a-b326-e69020dfc6e8" containerName="main"
Sep 10 00:07:07.533161 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.533127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.535802 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.535733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-8376bc-serving-cert\""
Sep 10 00:07:07.540463 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.540437    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:07:07.670998 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.670965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.670998 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.670998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.671198 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.671031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.671198 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.671058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krsgp\" (UniqueName: \"kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.772624 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.772599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.772624 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.772627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.772858 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.772659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.772858 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.772689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krsgp\" (UniqueName: \"kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.773207 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.773186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.775226 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.775197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.775226 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.775217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.780080 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.780033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krsgp\" (UniqueName: \"kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp\") pod \"maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:07.848948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:07.848919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:08.129714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.129687    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:07:08.131875 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:07:08.131843    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefcee55_3d71_413a_b150_d4b2fbf716b2.slice/crio-6634fe3669a33515d654bf0023091e9f86ca5684b2dc48d23fc707ffb9513cd8 WatchSource:0}: Error finding container 6634fe3669a33515d654bf0023091e9f86ca5684b2dc48d23fc707ffb9513cd8: Status 404 returned error can't find the container with id 6634fe3669a33515d654bf0023091e9f86ca5684b2dc48d23fc707ffb9513cd8
Sep 10 00:07:08.239814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.239774    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:07:08.244663 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.244641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.247076 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.247055    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8376bc\""
Sep 10 00:07:08.253705 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.253684    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:07:08.339889 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.339856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" event={"ID":"defcee55-3d71-413a-b150-d4b2fbf716b2","Type":"ContainerStarted","Data":"6634fe3669a33515d654bf0023091e9f86ca5684b2dc48d23fc707ffb9513cd8"}
Sep 10 00:07:08.379360 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.379283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjmg\" (UniqueName: \"kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.379360 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.379346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.480509 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.480481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.480641 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.480564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjmg\" (UniqueName: \"kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.481180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.481157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.488742 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.488717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjmg\" (UniqueName: \"kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg\") pod \"payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.556893 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.556863    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:08.830252 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:08.830220    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:07:08.831126 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:07:08.831100    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0733411_6e2a_435f_b78e_c799d7df3868.slice/crio-fe3ccf695b44526d198d3e829e9620537cd17c81a8c3f1a6705b95d66164305a WatchSource:0}: Error finding container fe3ccf695b44526d198d3e829e9620537cd17c81a8c3f1a6705b95d66164305a: Status 404 returned error can't find the container with id fe3ccf695b44526d198d3e829e9620537cd17c81a8c3f1a6705b95d66164305a
Sep 10 00:07:09.347221 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:09.347187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" event={"ID":"f0733411-6e2a-435f-b78e-c799d7df3868","Type":"ContainerStarted","Data":"29f759f7e30ae1a23f6a5ffa3f557cf5e21635dc95285ea17f90519ab8259518"}
Sep 10 00:07:09.347705 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:09.347240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" event={"ID":"f0733411-6e2a-435f-b78e-c799d7df3868","Type":"ContainerStarted","Data":"fe3ccf695b44526d198d3e829e9620537cd17c81a8c3f1a6705b95d66164305a"}
Sep 10 00:07:09.347705 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:09.347269    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:09.362353 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:09.362313    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" podStartSLOduration=1.362298205 podStartE2EDuration="1.362298205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:09.361760689 +0000 UTC m=+1102.682136752" watchObservedRunningTime="2026-09-10 00:07:09.362298205 +0000 UTC m=+1102.682674268"
Sep 10 00:07:10.354666 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:10.354626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" event={"ID":"defcee55-3d71-413a-b150-d4b2fbf716b2","Type":"ContainerStarted","Data":"a4598406994e7dfbb7e2faba41b82b041e419c58b057705924864f4e86989957"}
Sep 10 00:07:10.355066 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:10.354705    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:10.375216 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:10.375160    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" podStartSLOduration=1.849833368 podStartE2EDuration="3.375143606s" totalImagesPullingTime="1.525310238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:10.3740037 +0000 UTC m=+1103.694379799" watchObservedRunningTime="2026-09-10 00:07:10.375143606 +0000 UTC m=+1103.695519668"
Sep 10 00:07:16.366127 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:16.366098    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:07:20.356623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:20.356593    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:07:39.350052 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.350019    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:07:39.353709 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.353692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.356012 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.355991    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a6d968\"/\"openshift-service-ca.crt\""
Sep 10 00:07:39.356898 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.356880    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a6d968\"/\"kube-root-ca.crt\""
Sep 10 00:07:39.356995 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.356933    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a6d968\"/\"test-model-a6d968-kserve-self-signed-certs\""
Sep 10 00:07:39.357062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.357018    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a6d968\"/\"default-dockercfg-64km8\""
Sep 10 00:07:39.360787 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.360763    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:07:39.473556 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.473711 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.473711 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.473791 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4j4\" (UniqueName: \"kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.473826 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.473826 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.473810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574442 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574442 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574735 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4j4\" (UniqueName: \"kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574735 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574735 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574735 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574889    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.574949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.575051 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.574990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.576871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.576851    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.577132 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.577112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.582554 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.582514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4j4\" (UniqueName: \"kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4\") pod \"test-model-a6d968-kserve-78c566c897-5nfnp\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") " pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.664074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.663982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:39.938762 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:39.938735    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:07:39.940397 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:07:39.940371    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b1eec63_27f3_452c_ac0a_533d7294927c.slice/crio-eef6f5ec20f2bd246d5cdf4f57a27bd062f7ad6023d0a342b8a2c1e601e6b6e5 WatchSource:0}: Error finding container eef6f5ec20f2bd246d5cdf4f57a27bd062f7ad6023d0a342b8a2c1e601e6b6e5: Status 404 returned error can't find the container with id eef6f5ec20f2bd246d5cdf4f57a27bd062f7ad6023d0a342b8a2c1e601e6b6e5
Sep 10 00:07:40.528497 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:40.528459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" event={"ID":"1b1eec63-27f3-452c-ac0a-533d7294927c","Type":"ContainerStarted","Data":"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"}
Sep 10 00:07:40.528925 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:40.528513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" event={"ID":"1b1eec63-27f3-452c-ac0a-533d7294927c","Type":"ContainerStarted","Data":"eef6f5ec20f2bd246d5cdf4f57a27bd062f7ad6023d0a342b8a2c1e601e6b6e5"}
Sep 10 00:07:40.546816 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:40.546763    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" podStartSLOduration=1.546746432 podStartE2EDuration="1.546746432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:40.543600488 +0000 UTC m=+1133.863976553" watchObservedRunningTime="2026-09-10 00:07:40.546746432 +0000 UTC m=+1133.867122495"
Sep 10 00:07:49.665038 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:49.665006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:49.665579 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:49.665060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:49.677560 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:49.677520    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:50.595786 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:50.595759    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:07:58.858583 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.858548    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:07:58.863276 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.863253    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.865962 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.865935    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8376bc\"/\"kube-root-ca.crt\""
Sep 10 00:07:58.867123 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.867036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8376bc\"/\"default-dockercfg-8dlbj\""
Sep 10 00:07:58.867280 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.867120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8376bc\"/\"explicit-model-8376bc-kserve-self-signed-certs\""
Sep 10 00:07:58.867492 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.867464    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8376bc\"/\"openshift-service-ca.crt\""
Sep 10 00:07:58.871242 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.871165    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:07:58.952552 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.952729 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.952729 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.952729 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whsr\" (UniqueName: \"kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.952842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952770    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:58.952842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:58.952810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whsr\" (UniqueName: \"kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.055973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.056608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.056198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.060111 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.060056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.060327 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.060301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.062255 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.062235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.067742 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.067720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whsr\" (UniqueName: \"kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr\") pod \"explicit-model-8376bc-kserve-578846c5ff-djc98\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") " pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.174890 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.174804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:07:59.450276 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.450252    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:07:59.451684 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:07:59.451656    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00988df7_fd5a_430d_a30e_b18ebdc45b0c.slice/crio-6fc700cfe8cfabd2867051761fcca702ad6127c82efe5e18e8bb4805a7b6c1b2 WatchSource:0}: Error finding container 6fc700cfe8cfabd2867051761fcca702ad6127c82efe5e18e8bb4805a7b6c1b2: Status 404 returned error can't find the container with id 6fc700cfe8cfabd2867051761fcca702ad6127c82efe5e18e8bb4805a7b6c1b2
Sep 10 00:07:59.636216 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.636178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" event={"ID":"00988df7-fd5a-430d-a30e-b18ebdc45b0c","Type":"ContainerStarted","Data":"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"}
Sep 10 00:07:59.636387 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.636229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" event={"ID":"00988df7-fd5a-430d-a30e-b18ebdc45b0c","Type":"ContainerStarted","Data":"6fc700cfe8cfabd2867051761fcca702ad6127c82efe5e18e8bb4805a7b6c1b2"}
Sep 10 00:07:59.653228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:07:59.653183    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" podStartSLOduration=1.6531673850000002 podStartE2EDuration="1.653167385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:59.652868503 +0000 UTC m=+1152.973244567" watchObservedRunningTime="2026-09-10 00:07:59.653167385 +0000 UTC m=+1152.973543449"
Sep 10 00:08:09.175234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:09.175150    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:09.175234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:09.175209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:09.187815 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:09.187788    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:09.706999 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:09.706964    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:37.147591 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:37.147533    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:08:43.606952 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.606916    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:08:43.607616 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.607183    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" podUID="defcee55-3d71-413a-b150-d4b2fbf716b2" containerName="maas-api" containerID="cri-o://a4598406994e7dfbb7e2faba41b82b041e419c58b057705924864f4e86989957" gracePeriod=30
Sep 10 00:08:43.667871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.667831    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:08:43.668164 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.668110    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" podUID="f0733411-6e2a-435f-b78e-c799d7df3868" containerName="payload-pre-processing" containerID="cri-o://29f759f7e30ae1a23f6a5ffa3f557cf5e21635dc95285ea17f90519ab8259518" gracePeriod=30
Sep 10 00:08:43.893180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.893147    2576 generic.go:356] "Generic (PLEG): container finished" podID="f0733411-6e2a-435f-b78e-c799d7df3868" containerID="29f759f7e30ae1a23f6a5ffa3f557cf5e21635dc95285ea17f90519ab8259518" exitCode=0
Sep 10 00:08:43.893382 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.893223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" event={"ID":"f0733411-6e2a-435f-b78e-c799d7df3868","Type":"ContainerDied","Data":"29f759f7e30ae1a23f6a5ffa3f557cf5e21635dc95285ea17f90519ab8259518"}
Sep 10 00:08:43.895382 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.895356    2576 generic.go:356] "Generic (PLEG): container finished" podID="defcee55-3d71-413a-b150-d4b2fbf716b2" containerID="a4598406994e7dfbb7e2faba41b82b041e419c58b057705924864f4e86989957" exitCode=0
Sep 10 00:08:43.895382 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.895375    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" event={"ID":"defcee55-3d71-413a-b150-d4b2fbf716b2","Type":"ContainerDied","Data":"a4598406994e7dfbb7e2faba41b82b041e419c58b057705924864f4e86989957"}
Sep 10 00:08:43.928378 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.928350    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:08:43.957351 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.957320    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:08:43.988004 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.987975    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume\") pod \"f0733411-6e2a-435f-b78e-c799d7df3868\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") "
Sep 10 00:08:43.988188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls\") pod \"defcee55-3d71-413a-b150-d4b2fbf716b2\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") "
Sep 10 00:08:43.988188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs\") pod \"defcee55-3d71-413a-b150-d4b2fbf716b2\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") "
Sep 10 00:08:43.988188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988130    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjmg\" (UniqueName: \"kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg\") pod \"f0733411-6e2a-435f-b78e-c799d7df3868\" (UID: \"f0733411-6e2a-435f-b78e-c799d7df3868\") "
Sep 10 00:08:43.988188 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988155    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca\") pod \"defcee55-3d71-413a-b150-d4b2fbf716b2\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") "
Sep 10 00:08:43.988398 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krsgp\" (UniqueName: \"kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp\") pod \"defcee55-3d71-413a-b150-d4b2fbf716b2\" (UID: \"defcee55-3d71-413a-b150-d4b2fbf716b2\") "
Sep 10 00:08:43.988398 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988366    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0733411-6e2a-435f-b78e-c799d7df3868" (UID: "f0733411-6e2a-435f-b78e-c799d7df3868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:08:43.988673 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "defcee55-3d71-413a-b150-d4b2fbf716b2" (UID: "defcee55-3d71-413a-b150-d4b2fbf716b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:08:43.988817 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988771    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0733411-6e2a-435f-b78e-c799d7df3868-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:43.988917 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.988857    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/defcee55-3d71-413a-b150-d4b2fbf716b2-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:43.990472 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.990443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "defcee55-3d71-413a-b150-d4b2fbf716b2" (UID: "defcee55-3d71-413a-b150-d4b2fbf716b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:08:43.990624 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.990557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp" (OuterVolumeSpecName: "kube-api-access-krsgp") pod "defcee55-3d71-413a-b150-d4b2fbf716b2" (UID: "defcee55-3d71-413a-b150-d4b2fbf716b2"). InnerVolumeSpecName "kube-api-access-krsgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:08:43.990804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.990783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "defcee55-3d71-413a-b150-d4b2fbf716b2" (UID: "defcee55-3d71-413a-b150-d4b2fbf716b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:08:43.990847 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:43.990826    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg" (OuterVolumeSpecName: "kube-api-access-xmjmg") pod "f0733411-6e2a-435f-b78e-c799d7df3868" (UID: "f0733411-6e2a-435f-b78e-c799d7df3868"). InnerVolumeSpecName "kube-api-access-xmjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:08:44.089403 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.089362    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjmg\" (UniqueName: \"kubernetes.io/projected/f0733411-6e2a-435f-b78e-c799d7df3868-kube-api-access-xmjmg\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.089403 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.089396    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krsgp\" (UniqueName: \"kubernetes.io/projected/defcee55-3d71-413a-b150-d4b2fbf716b2-kube-api-access-krsgp\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.089403 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.089409    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.089693 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.089420    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/defcee55-3d71-413a-b150-d4b2fbf716b2-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.445923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.445885    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:08:44.446276 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446264    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0733411-6e2a-435f-b78e-c799d7df3868" containerName="payload-pre-processing"
Sep 10 00:08:44.446322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446278    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0733411-6e2a-435f-b78e-c799d7df3868" containerName="payload-pre-processing"
Sep 10 00:08:44.446322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446286    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defcee55-3d71-413a-b150-d4b2fbf716b2" containerName="maas-api"
Sep 10 00:08:44.446322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446292    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="defcee55-3d71-413a-b150-d4b2fbf716b2" containerName="maas-api"
Sep 10 00:08:44.446414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446348    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0733411-6e2a-435f-b78e-c799d7df3868" containerName="payload-pre-processing"
Sep 10 00:08:44.446414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.446356    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="defcee55-3d71-413a-b150-d4b2fbf716b2" containerName="maas-api"
Sep 10 00:08:44.449376 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.449357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.451884 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.451856    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"distinct-w4-cc0040-kserve-self-signed-certs\""
Sep 10 00:08:44.457486 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.457463    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:08:44.492302 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.492302 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.492510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.492510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8ll\" (UniqueName: \"kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.492510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.492635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.492533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593042 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.592986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593042 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593052    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8ll\" (UniqueName: \"kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593590 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593590 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.593683 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.593592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.595674 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.595647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.595902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.595884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.601579 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.601534    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8ll\" (UniqueName: \"kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll\") pod \"distinct-w4-cc0040-kserve-54496455fc-ksh4k\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.760112 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.760014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:44.904344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.904317    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"
Sep 10 00:08:44.904344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.904336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq" event={"ID":"f0733411-6e2a-435f-b78e-c799d7df3868","Type":"ContainerDied","Data":"fe3ccf695b44526d198d3e829e9620537cd17c81a8c3f1a6705b95d66164305a"}
Sep 10 00:08:44.904789 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.904409    2576 scope.go:117] "RemoveContainer" containerID="29f759f7e30ae1a23f6a5ffa3f557cf5e21635dc95285ea17f90519ab8259518"
Sep 10 00:08:44.906037 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.905992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp" event={"ID":"defcee55-3d71-413a-b150-d4b2fbf716b2","Type":"ContainerDied","Data":"6634fe3669a33515d654bf0023091e9f86ca5684b2dc48d23fc707ffb9513cd8"}
Sep 10 00:08:44.906037 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.906001    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"
Sep 10 00:08:44.915843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.915684    2576 scope.go:117] "RemoveContainer" containerID="a4598406994e7dfbb7e2faba41b82b041e419c58b057705924864f4e86989957"
Sep 10 00:08:44.930483 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.930449    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:08:44.939688 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.939664    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8376bc-775c647d86-phxpq"]
Sep 10 00:08:44.949027 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.949003    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:08:44.952280 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:44.952257    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8376bc-74b5c99f55-gbhkp"]
Sep 10 00:08:45.037559 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.037517    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:08:45.038449 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:08:45.038428    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67bedfda_a6d7_4480_b3f8_6854f6977975.slice/crio-01d06fa085ae108505dbeb949dbe32986620df9ec43f5ffa1e4f72ca881ebd6f WatchSource:0}: Error finding container 01d06fa085ae108505dbeb949dbe32986620df9ec43f5ffa1e4f72ca881ebd6f: Status 404 returned error can't find the container with id 01d06fa085ae108505dbeb949dbe32986620df9ec43f5ffa1e4f72ca881ebd6f
Sep 10 00:08:45.204255 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.204219    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defcee55-3d71-413a-b150-d4b2fbf716b2" path="/var/lib/kubelet/pods/defcee55-3d71-413a-b150-d4b2fbf716b2/volumes"
Sep 10 00:08:45.204615 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.204602    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0733411-6e2a-435f-b78e-c799d7df3868" path="/var/lib/kubelet/pods/f0733411-6e2a-435f-b78e-c799d7df3868/volumes"
Sep 10 00:08:45.920606 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.920569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" event={"ID":"67bedfda-a6d7-4480-b3f8-6854f6977975","Type":"ContainerStarted","Data":"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"}
Sep 10 00:08:45.921091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.920617    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" event={"ID":"67bedfda-a6d7-4480-b3f8-6854f6977975","Type":"ContainerStarted","Data":"01d06fa085ae108505dbeb949dbe32986620df9ec43f5ffa1e4f72ca881ebd6f"}
Sep 10 00:08:45.938211 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:45.938161    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" podStartSLOduration=1.9381449800000001 podStartE2EDuration="1.93814498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:08:45.935044303 +0000 UTC m=+1199.255420368" watchObservedRunningTime="2026-09-10 00:08:45.93814498 +0000 UTC m=+1199.258521086"
Sep 10 00:08:49.038423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.038383    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:08:49.043426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.043405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.046116 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.046092    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb0fe265\""
Sep 10 00:08:49.046234 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.046217    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb0fe265-dockercfg-d8flb\""
Sep 10 00:08:49.055785 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.055765    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:08:49.134991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.134936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkgq\" (UniqueName: \"kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.135170 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.135032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.236240 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.236206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkgq\" (UniqueName: \"kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.236240 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.236254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.237011 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.236991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.243852 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.243829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkgq\" (UniqueName: \"kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq\") pod \"payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.353626 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.353595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.629477 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.629448    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:08:49.630747 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:08:49.630721    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c0e48c_3c97_42e1_a7b1_a533b9a3f7ec.slice/crio-5535a8b13915d67d2351c7cc2254979879e91561671abad6bb2f73ad7ac46dc9 WatchSource:0}: Error finding container 5535a8b13915d67d2351c7cc2254979879e91561671abad6bb2f73ad7ac46dc9: Status 404 returned error can't find the container with id 5535a8b13915d67d2351c7cc2254979879e91561671abad6bb2f73ad7ac46dc9
Sep 10 00:08:49.945819 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.945713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" event={"ID":"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec","Type":"ContainerStarted","Data":"a18766a937942bb956561a3f191af808ca2170ec3293f06a96493dd186bb6761"}
Sep 10 00:08:49.945819 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.945770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" event={"ID":"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec","Type":"ContainerStarted","Data":"5535a8b13915d67d2351c7cc2254979879e91561671abad6bb2f73ad7ac46dc9"}
Sep 10 00:08:49.946074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.945892    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:08:49.962598 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:49.962553    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" podStartSLOduration=0.962523228 podStartE2EDuration="962.523228ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:08:49.960482838 +0000 UTC m=+1203.280858915" watchObservedRunningTime="2026-09-10 00:08:49.962523228 +0000 UTC m=+1203.282899292"
Sep 10 00:08:52.168042 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.167999    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" podUID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" containerName="main" containerID="cri-o://fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1" gracePeriod=45
Sep 10 00:08:52.421492 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.421439    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:52.467045 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467016    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467182 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467182 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467096    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467182 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467138    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2whsr\" (UniqueName: \"kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467182 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467206    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm\") pod \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\" (UID: \"00988df7-fd5a-430d-a30e-b18ebdc45b0c\") "
Sep 10 00:08:52.467397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467346    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:08:52.467510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:08:52.467594 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home" (OuterVolumeSpecName: "home") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:08:52.467646 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467617    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.467646 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.467636    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.469494 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.469461    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr" (OuterVolumeSpecName: "kube-api-access-2whsr") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "kube-api-access-2whsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:08:52.469635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.469498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:08:52.469635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.469512    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm" (OuterVolumeSpecName: "dshm") pod "00988df7-fd5a-430d-a30e-b18ebdc45b0c" (UID: "00988df7-fd5a-430d-a30e-b18ebdc45b0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:08:52.568167 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.568120    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whsr\" (UniqueName: \"kubernetes.io/projected/00988df7-fd5a-430d-a30e-b18ebdc45b0c-kube-api-access-2whsr\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.568167 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.568161    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.568167 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.568172    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00988df7-fd5a-430d-a30e-b18ebdc45b0c-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.568383 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.568179    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00988df7-fd5a-430d-a30e-b18ebdc45b0c-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:08:52.964478 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.964449    2576 generic.go:356] "Generic (PLEG): container finished" podID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" containerID="fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1" exitCode=0
Sep 10 00:08:52.964669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.964492    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"
Sep 10 00:08:52.964669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.964493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" event={"ID":"00988df7-fd5a-430d-a30e-b18ebdc45b0c","Type":"ContainerDied","Data":"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"}
Sep 10 00:08:52.964669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.964585    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98" event={"ID":"00988df7-fd5a-430d-a30e-b18ebdc45b0c","Type":"ContainerDied","Data":"6fc700cfe8cfabd2867051761fcca702ad6127c82efe5e18e8bb4805a7b6c1b2"}
Sep 10 00:08:52.964669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.964603    2576 scope.go:117] "RemoveContainer" containerID="fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"
Sep 10 00:08:52.974571 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.974530    2576 scope.go:117] "RemoveContainer" containerID="fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"
Sep 10 00:08:52.974838 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:08:52.974821    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1\": container with ID starting with fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1 not found: ID does not exist" containerID="fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"
Sep 10 00:08:52.974891 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.974847    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1"} err="failed to get container status \"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1\": rpc error: code = NotFound desc = could not find container \"fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1\": container with ID starting with fac9f3691f9de5eb0b1030a618a65da770a3f037d1f8bffa65f4795d0ff909e1 not found: ID does not exist"
Sep 10 00:08:52.985075 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.985037    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:08:52.990658 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:52.990639    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8376bc/explicit-model-8376bc-kserve-578846c5ff-djc98"]
Sep 10 00:08:53.202641 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:53.202602    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" path="/var/lib/kubelet/pods/00988df7-fd5a-430d-a30e-b18ebdc45b0c/volumes"
Sep 10 00:08:54.761043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:54.761005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:54.761490 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:54.761054    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:54.773757 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:54.773730    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:08:54.988252 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:08:54.988223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:09:00.953747 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:00.953717    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:09:29.933716 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:29.933685    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:09:29.936733 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:29.933930    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" podUID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" containerName="payload-pre-processing" containerID="cri-o://a18766a937942bb956561a3f191af808ca2170ec3293f06a96493dd186bb6761" gracePeriod=30
Sep 10 00:09:30.173146 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.173112    2576 generic.go:356] "Generic (PLEG): container finished" podID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" containerID="a18766a937942bb956561a3f191af808ca2170ec3293f06a96493dd186bb6761" exitCode=0
Sep 10 00:09:30.173299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.173181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" event={"ID":"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec","Type":"ContainerDied","Data":"a18766a937942bb956561a3f191af808ca2170ec3293f06a96493dd186bb6761"}
Sep 10 00:09:30.195751 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.195693    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:09:30.221090 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.221056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfkgq\" (UniqueName: \"kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq\") pod \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") "
Sep 10 00:09:30.221264 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.221105    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume\") pod \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\" (UID: \"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec\") "
Sep 10 00:09:30.221516 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.221488    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" (UID: "02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:09:30.223259 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.223237    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq" (OuterVolumeSpecName: "kube-api-access-hfkgq") pod "02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" (UID: "02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec"). InnerVolumeSpecName "kube-api-access-hfkgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:09:30.322322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.322289    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfkgq\" (UniqueName: \"kubernetes.io/projected/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-kube-api-access-hfkgq\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:09:30.322322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:30.322315    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:09:31.180727 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:31.180695    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"
Sep 10 00:09:31.181220 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:31.180691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l" event={"ID":"02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec","Type":"ContainerDied","Data":"5535a8b13915d67d2351c7cc2254979879e91561671abad6bb2f73ad7ac46dc9"}
Sep 10 00:09:31.181220 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:31.180849    2576 scope.go:117] "RemoveContainer" containerID="a18766a937942bb956561a3f191af808ca2170ec3293f06a96493dd186bb6761"
Sep 10 00:09:31.203029 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:31.202999    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:09:31.205443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:31.205421    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb0fe265-5b66d7b95d-wlp5l"]
Sep 10 00:09:33.203518 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:33.203481    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" path="/var/lib/kubelet/pods/02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec/volumes"
Sep 10 00:09:49.357752 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.357716    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:09:49.358121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358076    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" containerName="payload-pre-processing"
Sep 10 00:09:49.358121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" containerName="payload-pre-processing"
Sep 10 00:09:49.358121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358113    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" containerName="main"
Sep 10 00:09:49.358121 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358120    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" containerName="main"
Sep 10 00:09:49.358266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358176    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c0e48c-3c97-42e1-a7b1-a533b9a3f7ec" containerName="payload-pre-processing"
Sep 10 00:09:49.358266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.358184    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00988df7-fd5a-430d-a30e-b18ebdc45b0c" containerName="main"
Sep 10 00:09:49.362364 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.362347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.365787 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.365764    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"openshift-service-ca.crt\""
Sep 10 00:09:49.365927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.365767    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"default-dockercfg-hkwkb\""
Sep 10 00:09:49.365927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.365794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"kube-root-ca.crt\""
Sep 10 00:09:49.366046 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.365824    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"distinct-2-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:09:49.371017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.370993    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:09:49.505624 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.505797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4k7\" (UniqueName: \"kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.505797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.505797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.505797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.505939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.505856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606782 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4k7\" (UniqueName: \"kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.606982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.606944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.607325 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.607176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.607325 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.607225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.607325 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.607285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.609354 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.609283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.609569 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.609534    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.615089 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.615056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4k7\" (UniqueName: \"kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7\") pod \"distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.673560 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.673511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:49.951172 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:49.951148    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:09:49.953616 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:09:49.953589    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1753370f_3aec_4b76_ba84_1add86a8002e.slice/crio-b76c41d2cee2bc3652f8105dc28f10923b17c8fa1c1d6ea0750ca5a2be88e261 WatchSource:0}: Error finding container b76c41d2cee2bc3652f8105dc28f10923b17c8fa1c1d6ea0750ca5a2be88e261: Status 404 returned error can't find the container with id b76c41d2cee2bc3652f8105dc28f10923b17c8fa1c1d6ea0750ca5a2be88e261
Sep 10 00:09:50.293972 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:50.293880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" event={"ID":"1753370f-3aec-4b76-ba84-1add86a8002e","Type":"ContainerStarted","Data":"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"}
Sep 10 00:09:50.293972 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:50.293918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" event={"ID":"1753370f-3aec-4b76-ba84-1add86a8002e","Type":"ContainerStarted","Data":"b76c41d2cee2bc3652f8105dc28f10923b17c8fa1c1d6ea0750ca5a2be88e261"}
Sep 10 00:09:50.310407 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:50.310361    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" podStartSLOduration=1.310346872 podStartE2EDuration="1.310346872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:09:50.309341377 +0000 UTC m=+1263.629717440" watchObservedRunningTime="2026-09-10 00:09:50.310346872 +0000 UTC m=+1263.630722936"
Sep 10 00:09:55.341600 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.341564    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:09:55.345190 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.345174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.347686 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.347667    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1d36d9\""
Sep 10 00:09:55.347783 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.347713    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1d36d9-dockercfg-w5mfk\""
Sep 10 00:09:55.356804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.356775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:09:55.418620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.418580    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:09:55.422112 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.422090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.432284 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.432260    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:09:55.467696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.467657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxb72\" (UniqueName: \"kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.467843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.467735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.568532 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.568501    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxb72\" (UniqueName: \"kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.568742 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.568579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.568742 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.568608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.568864 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.568738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gxq\" (UniqueName: \"kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.569349 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.569331    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.576609 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.576593    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxb72\" (UniqueName: \"kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72\") pod \"payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.657760 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.657671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:55.670227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.670202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gxq\" (UniqueName: \"kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.670324 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.670267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.670888 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.670871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.678447 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.678415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gxq\" (UniqueName: \"kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq\") pod \"payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.733448 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.733403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:55.931418 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:55.931381    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:09:55.932559 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:09:55.932515    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3c2cc_a437_4135_a8e8_070eeabec58a.slice/crio-e5d4f818293162fcdfe85bc35ec6a7c04904a1e2c3143602e670d4766469b796 WatchSource:0}: Error finding container e5d4f818293162fcdfe85bc35ec6a7c04904a1e2c3143602e670d4766469b796: Status 404 returned error can't find the container with id e5d4f818293162fcdfe85bc35ec6a7c04904a1e2c3143602e670d4766469b796
Sep 10 00:09:56.009946 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.009921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:09:56.011658 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:09:56.011632    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ebc238_edf1_4c4e_bf97_cd4551887c6b.slice/crio-69fe76f059f865cb88d093ea2c70352342a6c28226d9947234173714f34574f2 WatchSource:0}: Error finding container 69fe76f059f865cb88d093ea2c70352342a6c28226d9947234173714f34574f2: Status 404 returned error can't find the container with id 69fe76f059f865cb88d093ea2c70352342a6c28226d9947234173714f34574f2
Sep 10 00:09:56.329137 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.329103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" event={"ID":"b1d3c2cc-a437-4135-a8e8-070eeabec58a","Type":"ContainerStarted","Data":"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"}
Sep 10 00:09:56.329351 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.329167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" event={"ID":"b1d3c2cc-a437-4135-a8e8-070eeabec58a","Type":"ContainerStarted","Data":"e5d4f818293162fcdfe85bc35ec6a7c04904a1e2c3143602e670d4766469b796"}
Sep 10 00:09:56.329351 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.329197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:09:56.330891 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.330859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" event={"ID":"05ebc238-edf1-4c4e-bf97-cd4551887c6b","Type":"ContainerStarted","Data":"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"}
Sep 10 00:09:56.331017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.330905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" event={"ID":"05ebc238-edf1-4c4e-bf97-cd4551887c6b","Type":"ContainerStarted","Data":"69fe76f059f865cb88d093ea2c70352342a6c28226d9947234173714f34574f2"}
Sep 10 00:09:56.331017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.330963    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:09:56.346913 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.346867    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" podStartSLOduration=1.34685327 podStartE2EDuration="1.34685327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:09:56.343961307 +0000 UTC m=+1269.664337372" watchObservedRunningTime="2026-09-10 00:09:56.34685327 +0000 UTC m=+1269.667229333"
Sep 10 00:09:56.358424 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:56.358385    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" podStartSLOduration=1.358371334 podStartE2EDuration="1.358371334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:09:56.357631124 +0000 UTC m=+1269.678007200" watchObservedRunningTime="2026-09-10 00:09:56.358371334 +0000 UTC m=+1269.678747401"
Sep 10 00:09:59.674712 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:59.674671    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:59.675199 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:59.674736    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:09:59.687063 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:09:59.687038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:10:00.365039 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:00.365013    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:10:07.338068 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:07.338039    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:10:07.338489 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:07.338147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:10:20.938900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:20.938868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:10:23.590955 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.590914    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:10:23.591381 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.591129    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" podUID="b5badaec-ba32-49fb-a75b-a42ea854358c" containerName="maas-api" containerID="cri-o://a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269" gracePeriod=30
Sep 10 00:10:23.639608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.639574    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:10:23.639837 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.639810    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" podUID="063b8bec-0b47-42bf-827a-fa420de00dd8" containerName="payload-processing" containerID="cri-o://211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e" gracePeriod=30
Sep 10 00:10:23.896051 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.896028    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:10:23.921608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:23.921583    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:10:24.033829 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033797    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v6x5\" (UniqueName: \"kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5\") pod \"b5badaec-ba32-49fb-a75b-a42ea854358c\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") "
Sep 10 00:10:24.034002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca\") pod \"b5badaec-ba32-49fb-a75b-a42ea854358c\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") "
Sep 10 00:10:24.034002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume\") pod \"063b8bec-0b47-42bf-827a-fa420de00dd8\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") "
Sep 10 00:10:24.034002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033919    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls\") pod \"b5badaec-ba32-49fb-a75b-a42ea854358c\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") "
Sep 10 00:10:24.034002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033955    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29bc4\" (UniqueName: \"kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4\") pod \"063b8bec-0b47-42bf-827a-fa420de00dd8\" (UID: \"063b8bec-0b47-42bf-827a-fa420de00dd8\") "
Sep 10 00:10:24.034002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.033999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs\") pod \"b5badaec-ba32-49fb-a75b-a42ea854358c\" (UID: \"b5badaec-ba32-49fb-a75b-a42ea854358c\") "
Sep 10 00:10:24.034333 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.034225    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b5badaec-ba32-49fb-a75b-a42ea854358c" (UID: "b5badaec-ba32-49fb-a75b-a42ea854358c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:24.034333 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.034298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "063b8bec-0b47-42bf-827a-fa420de00dd8" (UID: "063b8bec-0b47-42bf-827a-fa420de00dd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:24.034454 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.034352    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5badaec-ba32-49fb-a75b-a42ea854358c-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.036198 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.036175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5" (OuterVolumeSpecName: "kube-api-access-4v6x5") pod "b5badaec-ba32-49fb-a75b-a42ea854358c" (UID: "b5badaec-ba32-49fb-a75b-a42ea854358c"). InnerVolumeSpecName "kube-api-access-4v6x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:24.036291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.036205    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4" (OuterVolumeSpecName: "kube-api-access-29bc4") pod "063b8bec-0b47-42bf-827a-fa420de00dd8" (UID: "063b8bec-0b47-42bf-827a-fa420de00dd8"). InnerVolumeSpecName "kube-api-access-29bc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:24.036291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.036242    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b5badaec-ba32-49fb-a75b-a42ea854358c" (UID: "b5badaec-ba32-49fb-a75b-a42ea854358c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:24.036291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.036268    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b5badaec-ba32-49fb-a75b-a42ea854358c" (UID: "b5badaec-ba32-49fb-a75b-a42ea854358c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:24.135010 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.134928    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6x5\" (UniqueName: \"kubernetes.io/projected/b5badaec-ba32-49fb-a75b-a42ea854358c-kube-api-access-4v6x5\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.135010 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.134957    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063b8bec-0b47-42bf-827a-fa420de00dd8-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.135010 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.134967    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.135010 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.134975    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29bc4\" (UniqueName: \"kubernetes.io/projected/063b8bec-0b47-42bf-827a-fa420de00dd8-kube-api-access-29bc4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.135010 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.134983    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5badaec-ba32-49fb-a75b-a42ea854358c-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.489104 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.489016    2576 generic.go:356] "Generic (PLEG): container finished" podID="b5badaec-ba32-49fb-a75b-a42ea854358c" containerID="a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269" exitCode=0
Sep 10 00:10:24.489104 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.489068    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"
Sep 10 00:10:24.489104 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.489096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" event={"ID":"b5badaec-ba32-49fb-a75b-a42ea854358c","Type":"ContainerDied","Data":"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"}
Sep 10 00:10:24.489391 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.489149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn" event={"ID":"b5badaec-ba32-49fb-a75b-a42ea854358c","Type":"ContainerDied","Data":"a487b64dee71fd145d2fef307a360d06f0a4e757cdd3533baa12a32e820da101"}
Sep 10 00:10:24.489391 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.489179    2576 scope.go:117] "RemoveContainer" containerID="a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"
Sep 10 00:10:24.490561 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.490515    2576 generic.go:356] "Generic (PLEG): container finished" podID="063b8bec-0b47-42bf-827a-fa420de00dd8" containerID="211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e" exitCode=0
Sep 10 00:10:24.490677 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.490574    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"
Sep 10 00:10:24.490677 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.490579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" event={"ID":"063b8bec-0b47-42bf-827a-fa420de00dd8","Type":"ContainerDied","Data":"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"}
Sep 10 00:10:24.490677 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.490629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw" event={"ID":"063b8bec-0b47-42bf-827a-fa420de00dd8","Type":"ContainerDied","Data":"8f210a14eb6925c90a41b9bf72446ec4e4f3cd16103560304f18af8db3c75309"}
Sep 10 00:10:24.501433 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.501413    2576 scope.go:117] "RemoveContainer" containerID="a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"
Sep 10 00:10:24.501842 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:10:24.501811    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269\": container with ID starting with a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269 not found: ID does not exist" containerID="a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"
Sep 10 00:10:24.501951 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.501851    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269"} err="failed to get container status \"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269\": rpc error: code = NotFound desc = could not find container \"a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269\": container with ID starting with a5b6df67c46faf34a622dbd95e2e14d00b4c3876bcbb6ba27a944b2b68cc6269 not found: ID does not exist"
Sep 10 00:10:24.501951 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.501875    2576 scope.go:117] "RemoveContainer" containerID="211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"
Sep 10 00:10:24.511244 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.511223    2576 scope.go:117] "RemoveContainer" containerID="211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"
Sep 10 00:10:24.511573 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:10:24.511550    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e\": container with ID starting with 211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e not found: ID does not exist" containerID="211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"
Sep 10 00:10:24.511655 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.511584    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e"} err="failed to get container status \"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e\": rpc error: code = NotFound desc = could not find container \"211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e\": container with ID starting with 211fdba084a96d83eca5b02f3080f4780eb4346a9b1f94d0c281e44333544f8e not found: ID does not exist"
Sep 10 00:10:24.513253 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.513235    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:10:24.517422 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.517401    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0f965-6d7df7f6b7-h66qn"]
Sep 10 00:10:24.527531 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.527510    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:10:24.531960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:24.531939    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0f965-5f649ddfcc-rfmqw"]
Sep 10 00:10:25.203681 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:25.203646    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063b8bec-0b47-42bf-827a-fa420de00dd8" path="/var/lib/kubelet/pods/063b8bec-0b47-42bf-827a-fa420de00dd8/volumes"
Sep 10 00:10:25.204148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:25.204015    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5badaec-ba32-49fb-a75b-a42ea854358c" path="/var/lib/kubelet/pods/b5badaec-ba32-49fb-a75b-a42ea854358c/volumes"
Sep 10 00:10:35.960564 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:35.960498    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" podUID="1b1eec63-27f3-452c-ac0a-533d7294927c" containerName="main" containerID="cri-o://63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690" gracePeriod=45
Sep 10 00:10:36.220608 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.220532    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:10:36.348965 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.348934    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349140 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349140 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349022    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349140 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349140 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349347 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349149    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x4j4\" (UniqueName: \"kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4\") pod \"1b1eec63-27f3-452c-ac0a-533d7294927c\" (UID: \"1b1eec63-27f3-452c-ac0a-533d7294927c\") "
Sep 10 00:10:36.349347 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:36.349347 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349303    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:36.349503 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349416    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home" (OuterVolumeSpecName: "home") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:36.349503 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349487    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.349627 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.349506    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.351340 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.351311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:36.351467 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.351351    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm" (OuterVolumeSpecName: "dshm") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:36.351467 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.351430    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4" (OuterVolumeSpecName: "kube-api-access-6x4j4") pod "1b1eec63-27f3-452c-ac0a-533d7294927c" (UID: "1b1eec63-27f3-452c-ac0a-533d7294927c"). InnerVolumeSpecName "kube-api-access-6x4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:36.450766 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.450739    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4j4\" (UniqueName: \"kubernetes.io/projected/1b1eec63-27f3-452c-ac0a-533d7294927c-kube-api-access-6x4j4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.450766 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.450762    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1eec63-27f3-452c-ac0a-533d7294927c-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.450909 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.450772    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.450909 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.450780    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1eec63-27f3-452c-ac0a-533d7294927c-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:36.563872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.563807    2576 generic.go:356] "Generic (PLEG): container finished" podID="1b1eec63-27f3-452c-ac0a-533d7294927c" containerID="63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690" exitCode=0
Sep 10 00:10:36.563872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.563849    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"
Sep 10 00:10:36.564020 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.563900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" event={"ID":"1b1eec63-27f3-452c-ac0a-533d7294927c","Type":"ContainerDied","Data":"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"}
Sep 10 00:10:36.564020 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.563943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp" event={"ID":"1b1eec63-27f3-452c-ac0a-533d7294927c","Type":"ContainerDied","Data":"eef6f5ec20f2bd246d5cdf4f57a27bd062f7ad6023d0a342b8a2c1e601e6b6e5"}
Sep 10 00:10:36.564020 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.563960    2576 scope.go:117] "RemoveContainer" containerID="63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"
Sep 10 00:10:36.573777 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.573761    2576 scope.go:117] "RemoveContainer" containerID="63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"
Sep 10 00:10:36.574025 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:10:36.574008    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690\": container with ID starting with 63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690 not found: ID does not exist" containerID="63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"
Sep 10 00:10:36.574079 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.574042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690"} err="failed to get container status \"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690\": rpc error: code = NotFound desc = could not find container \"63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690\": container with ID starting with 63f3975480dae506e6b08fc6326769ce1aaa23d414fbaeeeccc44f491829a690 not found: ID does not exist"
Sep 10 00:10:36.584322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.584300    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:10:36.588072 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:36.588052    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a6d968/test-model-a6d968-kserve-78c566c897-5nfnp"]
Sep 10 00:10:37.202997 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:37.202964    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1eec63-27f3-452c-ac0a-533d7294927c" path="/var/lib/kubelet/pods/1b1eec63-27f3-452c-ac0a-533d7294927c/volumes"
Sep 10 00:10:40.918029 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:40.917996    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:10:40.918412 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:40.918232    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" podUID="86592b8b-5214-4600-8d08-8c95468489a3" containerName="maas-api" containerID="cri-o://f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c" gracePeriod=30
Sep 10 00:10:41.160490 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.160465    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:10:41.295301 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.295213    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2hgn\" (UniqueName: \"kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn\") pod \"86592b8b-5214-4600-8d08-8c95468489a3\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") "
Sep 10 00:10:41.295301 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.295279    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca\") pod \"86592b8b-5214-4600-8d08-8c95468489a3\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") "
Sep 10 00:10:41.295528 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.295305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs\") pod \"86592b8b-5214-4600-8d08-8c95468489a3\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") "
Sep 10 00:10:41.295528 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.295348    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") pod \"86592b8b-5214-4600-8d08-8c95468489a3\" (UID: \"86592b8b-5214-4600-8d08-8c95468489a3\") "
Sep 10 00:10:41.295728 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.295684    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86592b8b-5214-4600-8d08-8c95468489a3" (UID: "86592b8b-5214-4600-8d08-8c95468489a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:41.297730 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.297701    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86592b8b-5214-4600-8d08-8c95468489a3" (UID: "86592b8b-5214-4600-8d08-8c95468489a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:41.297730 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.297709    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn" (OuterVolumeSpecName: "kube-api-access-w2hgn") pod "86592b8b-5214-4600-8d08-8c95468489a3" (UID: "86592b8b-5214-4600-8d08-8c95468489a3"). InnerVolumeSpecName "kube-api-access-w2hgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:41.297730 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.297716    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "86592b8b-5214-4600-8d08-8c95468489a3" (UID: "86592b8b-5214-4600-8d08-8c95468489a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:41.397072 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.397044    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86592b8b-5214-4600-8d08-8c95468489a3-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.397072 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.397067    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.397072 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.397076    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86592b8b-5214-4600-8d08-8c95468489a3-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.397296 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.397084    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hgn\" (UniqueName: \"kubernetes.io/projected/86592b8b-5214-4600-8d08-8c95468489a3-kube-api-access-w2hgn\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.593062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.593027    2576 generic.go:356] "Generic (PLEG): container finished" podID="86592b8b-5214-4600-8d08-8c95468489a3" containerID="f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c" exitCode=0
Sep 10 00:10:41.593227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.593077    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"
Sep 10 00:10:41.593227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.593107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" event={"ID":"86592b8b-5214-4600-8d08-8c95468489a3","Type":"ContainerDied","Data":"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"}
Sep 10 00:10:41.593227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.593160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4" event={"ID":"86592b8b-5214-4600-8d08-8c95468489a3","Type":"ContainerDied","Data":"182e85dfce412ca6e8e2beab117ed01d8773dd246f2d6b9c3332070884b61751"}
Sep 10 00:10:41.593227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.593181    2576 scope.go:117] "RemoveContainer" containerID="f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"
Sep 10 00:10:41.604097 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.604078    2576 scope.go:117] "RemoveContainer" containerID="f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"
Sep 10 00:10:41.604374 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:10:41.604358    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c\": container with ID starting with f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c not found: ID does not exist" containerID="f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"
Sep 10 00:10:41.604431 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.604381    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c"} err="failed to get container status \"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c\": rpc error: code = NotFound desc = could not find container \"f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c\": container with ID starting with f6ffa26ac97bdc7bd374ee325ed639fa4320b5f2f22d402959b36967406a172c not found: ID does not exist"
Sep 10 00:10:41.619552 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.619520    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:10:41.625096 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:41.625075    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a6d968-59d8966d5d-hkjw4"]
Sep 10 00:10:43.202443 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:43.202403    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86592b8b-5214-4600-8d08-8c95468489a3" path="/var/lib/kubelet/pods/86592b8b-5214-4600-8d08-8c95468489a3/volumes"
Sep 10 00:10:53.584523 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584488    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584879    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86592b8b-5214-4600-8d08-8c95468489a3" containerName="maas-api"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="86592b8b-5214-4600-8d08-8c95468489a3" containerName="maas-api"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584903    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063b8bec-0b47-42bf-827a-fa420de00dd8" containerName="payload-processing"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584909    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="063b8bec-0b47-42bf-827a-fa420de00dd8" containerName="payload-processing"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584916    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1eec63-27f3-452c-ac0a-533d7294927c" containerName="main"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584921    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1eec63-27f3-452c-ac0a-533d7294927c" containerName="main"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584931    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5badaec-ba32-49fb-a75b-a42ea854358c" containerName="maas-api"
Sep 10 00:10:53.584939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.584935    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5badaec-ba32-49fb-a75b-a42ea854358c" containerName="maas-api"
Sep 10 00:10:53.585184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.585007    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5badaec-ba32-49fb-a75b-a42ea854358c" containerName="maas-api"
Sep 10 00:10:53.585184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.585017    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="063b8bec-0b47-42bf-827a-fa420de00dd8" containerName="payload-processing"
Sep 10 00:10:53.585184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.585025    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1eec63-27f3-452c-ac0a-533d7294927c" containerName="main"
Sep 10 00:10:53.585184 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.585031    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="86592b8b-5214-4600-8d08-8c95468489a3" containerName="maas-api"
Sep 10 00:10:53.589746 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.589726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.592733 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.592708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1de9a7\""
Sep 10 00:10:53.592868 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.592840    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1de9a7-dockercfg-snd7v\""
Sep 10 00:10:53.598333 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.598289    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:10:53.602478 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.602444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.602610 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.602528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kfmv\" (UniqueName: \"kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.626278 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.626254    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:10:53.629657 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.629641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.638954 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.638927    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:10:53.703999 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.703971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.704159 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.704054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdxd\" (UniqueName: \"kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.704159 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.704080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.704159 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.704100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kfmv\" (UniqueName: \"kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.704743 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.704724    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.712144 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.712117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kfmv\" (UniqueName: \"kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv\") pod \"payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.804963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.804928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdxd\" (UniqueName: \"kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.804963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.804965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.805717 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.805699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.812508 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.812474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdxd\" (UniqueName: \"kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd\") pod \"payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:53.903433 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.903346    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:53.941141 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:53.941105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:54.182116 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.182089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:10:54.182945 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:10:54.182919    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27583020_b85b_4e47_8cd0_f1022277a989.slice/crio-ee916dd7f7a7eda0cbbbb440fd823d65486020c69544f715708447dd374455aa WatchSource:0}: Error finding container ee916dd7f7a7eda0cbbbb440fd823d65486020c69544f715708447dd374455aa: Status 404 returned error can't find the container with id ee916dd7f7a7eda0cbbbb440fd823d65486020c69544f715708447dd374455aa
Sep 10 00:10:54.226462 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.226434    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:10:54.226606 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:10:54.226518    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5e98b3_2202_4c64_a703_b4187bbe92ad.slice/crio-1a57898970b75bf51cfbf876886e0baa05a38e90cfc8421ac893936c469f2064 WatchSource:0}: Error finding container 1a57898970b75bf51cfbf876886e0baa05a38e90cfc8421ac893936c469f2064: Status 404 returned error can't find the container with id 1a57898970b75bf51cfbf876886e0baa05a38e90cfc8421ac893936c469f2064
Sep 10 00:10:54.669984 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.669950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" event={"ID":"27583020-b85b-4e47-8cd0-f1022277a989","Type":"ContainerStarted","Data":"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"}
Sep 10 00:10:54.670419 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.670001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" event={"ID":"27583020-b85b-4e47-8cd0-f1022277a989","Type":"ContainerStarted","Data":"ee916dd7f7a7eda0cbbbb440fd823d65486020c69544f715708447dd374455aa"}
Sep 10 00:10:54.670419 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.670042    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:10:54.671593 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.671567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" event={"ID":"9c5e98b3-2202-4c64-a703-b4187bbe92ad","Type":"ContainerStarted","Data":"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"}
Sep 10 00:10:54.671707 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.671609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" event={"ID":"9c5e98b3-2202-4c64-a703-b4187bbe92ad","Type":"ContainerStarted","Data":"1a57898970b75bf51cfbf876886e0baa05a38e90cfc8421ac893936c469f2064"}
Sep 10 00:10:54.671707 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.671627    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:10:54.686966 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.686910    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" podStartSLOduration=1.68689628 podStartE2EDuration="1.68689628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:54.68505895 +0000 UTC m=+1328.005435009" watchObservedRunningTime="2026-09-10 00:10:54.68689628 +0000 UTC m=+1328.007272344"
Sep 10 00:10:54.703830 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:54.703778    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" podStartSLOduration=1.7037619739999998 podStartE2EDuration="1.703761974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:54.700229772 +0000 UTC m=+1328.020605838" watchObservedRunningTime="2026-09-10 00:10:54.703761974 +0000 UTC m=+1328.024138046"
Sep 10 00:10:59.969739 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:59.969703    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:10:59.973699 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:59.973676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:10:59.976204 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:59.976179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-557790f7-serving-cert\""
Sep 10 00:10:59.983939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:10:59.983361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:11:00.066021 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.065988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.066164 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.066046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.066164 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.066074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.066242 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.066173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl6g\" (UniqueName: \"kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.167599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.167562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.167739 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.167662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.167739 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.167708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.167818 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.167755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl6g\" (UniqueName: \"kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.168306 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.168281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.170089 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.170062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.170254 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.170234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.175191 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.175167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl6g\" (UniqueName: \"kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g\") pod \"maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.288749 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.288676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:00.563900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.563874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:11:00.565369 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:11:00.565342    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc745fac_caf3_49f1_9ccb_559b99f5ef89.slice/crio-22b730414ce3f592a1d9a9d57dec48740c90f85590f20432949c6bb93539dbc5 WatchSource:0}: Error finding container 22b730414ce3f592a1d9a9d57dec48740c90f85590f20432949c6bb93539dbc5: Status 404 returned error can't find the container with id 22b730414ce3f592a1d9a9d57dec48740c90f85590f20432949c6bb93539dbc5
Sep 10 00:11:00.566833 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.566814    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:11:00.712978 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.712936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" event={"ID":"cc745fac-caf3-49f1-9ccb-559b99f5ef89","Type":"ContainerStarted","Data":"22b730414ce3f592a1d9a9d57dec48740c90f85590f20432949c6bb93539dbc5"}
Sep 10 00:11:00.714145 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.714120    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:11:00.718774 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.718757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.721421 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.721395    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-557790f7-dockercfg-dcsbg\""
Sep 10 00:11:00.721579 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.721432    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-557790f7\""
Sep 10 00:11:00.724133 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.724109    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:11:00.773335 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.773298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68rd\" (UniqueName: \"kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.773468 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.773358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.874634 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.874606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68rd\" (UniqueName: \"kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.874804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.874657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.875328 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.875309    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:00.881877 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:00.881858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68rd\" (UniqueName: \"kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd\") pod \"payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") " pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:01.032480 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.032453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:01.305240 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.305210    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:11:01.305388 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:11:01.305321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7cba04_6d05_44f9_9079_3674eba53686.slice/crio-ae2aaf3e674e3b45e8cede4f53d6a44a51eb58f91c3a65b8561c3ed6b6671b65 WatchSource:0}: Error finding container ae2aaf3e674e3b45e8cede4f53d6a44a51eb58f91c3a65b8561c3ed6b6671b65: Status 404 returned error can't find the container with id ae2aaf3e674e3b45e8cede4f53d6a44a51eb58f91c3a65b8561c3ed6b6671b65
Sep 10 00:11:01.720344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.720310    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" event={"ID":"7c7cba04-6d05-44f9-9079-3674eba53686","Type":"ContainerStarted","Data":"840d2f03866e4ec03b20c91574ca89ae6273e068b933a522d7e343b0417ebcb1"}
Sep 10 00:11:01.720573 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.720370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" event={"ID":"7c7cba04-6d05-44f9-9079-3674eba53686","Type":"ContainerStarted","Data":"ae2aaf3e674e3b45e8cede4f53d6a44a51eb58f91c3a65b8561c3ed6b6671b65"}
Sep 10 00:11:01.720573 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.720512    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:01.722091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.722067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" event={"ID":"cc745fac-caf3-49f1-9ccb-559b99f5ef89","Type":"ContainerStarted","Data":"868210aea81356090be2e47922139f9363369448d505e536edac264f0a00c7cd"}
Sep 10 00:11:01.722213 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.722161    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:01.737883 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.737845    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" podStartSLOduration=1.737832695 podStartE2EDuration="1.737832695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:01.735691609 +0000 UTC m=+1335.056067673" watchObservedRunningTime="2026-09-10 00:11:01.737832695 +0000 UTC m=+1335.058208759"
Sep 10 00:11:01.754100 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:01.754053    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" podStartSLOduration=2.364594564 podStartE2EDuration="2.754036589s" totalImagesPullingTime="389.442025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:01.750768807 +0000 UTC m=+1335.071144871" watchObservedRunningTime="2026-09-10 00:11:01.754036589 +0000 UTC m=+1335.074412654"
Sep 10 00:11:05.679714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:05.679682    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:11:05.680078 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:05.679852    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:11:07.733375 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:07.733345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:11:12.730768 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:12.730673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:11:21.889221 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.889185    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:11:21.892430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.892414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:21.894966 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.894942    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b1baac\""
Sep 10 00:11:21.895079 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.895018    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b1baac-dockercfg-x7n88\""
Sep 10 00:11:21.907425 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.907401    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:11:21.980621 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.980593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74zk\" (UniqueName: \"kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:21.980803 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:21.980706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.081936 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.081903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.082104 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.081974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74zk\" (UniqueName: \"kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.082649 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.082633    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.089776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.089746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74zk\" (UniqueName: \"kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk\") pod \"payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.202650 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.202566    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.474361 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.474332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:11:22.475378 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:11:22.475352    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587f16a6_9716_4431_88c2_2f5faa8acc41.slice/crio-e02a88dacd8aea8bdc01ae178e646cb77e077f26bc6bc78f886a424daec46c37 WatchSource:0}: Error finding container e02a88dacd8aea8bdc01ae178e646cb77e077f26bc6bc78f886a424daec46c37: Status 404 returned error can't find the container with id e02a88dacd8aea8bdc01ae178e646cb77e077f26bc6bc78f886a424daec46c37
Sep 10 00:11:22.847987 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.847959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" event={"ID":"587f16a6-9716-4431-88c2-2f5faa8acc41","Type":"ContainerStarted","Data":"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"}
Sep 10 00:11:22.848180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.848006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" event={"ID":"587f16a6-9716-4431-88c2-2f5faa8acc41","Type":"ContainerStarted","Data":"e02a88dacd8aea8bdc01ae178e646cb77e077f26bc6bc78f886a424daec46c37"}
Sep 10 00:11:22.848180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.848070    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:11:22.864946 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:22.864896    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" podStartSLOduration=1.864882996 podStartE2EDuration="1.864882996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:22.862289319 +0000 UTC m=+1356.182665385" watchObservedRunningTime="2026-09-10 00:11:22.864882996 +0000 UTC m=+1356.185259125"
Sep 10 00:11:33.855607 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:11:33.855578    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:12:10.745525 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.745488    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:12:10.750239 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.750222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.754015 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.753989    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-557790f7\"/\"default-dockercfg-9k7pr\""
Sep 10 00:12:10.754249 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.754230    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-557790f7\"/\"openshift-service-ca.crt\""
Sep 10 00:12:10.754513 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.754492    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-557790f7\"/\"kube-root-ca.crt\""
Sep 10 00:12:10.754635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.754492    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-557790f7\"/\"e2e-lifecycle-model-557790f7-kserve-self-signed-certs\""
Sep 10 00:12:10.758668 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.758643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:12:10.854017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.853983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.854017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.854021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.854235 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.854043    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6vh\" (UniqueName: \"kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.854235 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.854172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.854235 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.854214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.854366 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.854283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.954991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6vh\" (UniqueName: \"kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955293 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955509 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955509 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.955631 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.955507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.957672 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.957652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.957848 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.957832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:10.963297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:10.963275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6vh\" (UniqueName: \"kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh\") pod \"e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") " pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:11.061790 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:11.061689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:11.337411 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:11.337373    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:12:11.338257 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:12:11.338230    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb4568c3_2cee_4c1d_8dda_b95c474081b5.slice/crio-481aee7b9a9aa9f103ec5bb97a9139be7cbf4e526ee6e3d0046609a38c8ff406 WatchSource:0}: Error finding container 481aee7b9a9aa9f103ec5bb97a9139be7cbf4e526ee6e3d0046609a38c8ff406: Status 404 returned error can't find the container with id 481aee7b9a9aa9f103ec5bb97a9139be7cbf4e526ee6e3d0046609a38c8ff406
Sep 10 00:12:12.135494 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:12.135457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" event={"ID":"fb4568c3-2cee-4c1d-8dda-b95c474081b5","Type":"ContainerStarted","Data":"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"}
Sep 10 00:12:12.135494 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:12.135499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" event={"ID":"fb4568c3-2cee-4c1d-8dda-b95c474081b5","Type":"ContainerStarted","Data":"481aee7b9a9aa9f103ec5bb97a9139be7cbf4e526ee6e3d0046609a38c8ff406"}
Sep 10 00:12:12.151148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:12.151099    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" podStartSLOduration=2.151082091 podStartE2EDuration="2.151082091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:12:12.150762814 +0000 UTC m=+1405.471138880" watchObservedRunningTime="2026-09-10 00:12:12.151082091 +0000 UTC m=+1405.471458155"
Sep 10 00:12:21.062053 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:21.062020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:21.062452 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:21.062071    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:21.074751 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:21.074723    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:21.205057 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:21.205029    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:12:30.869154 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.869120    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:12:30.872761 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.872745    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.875384 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.875360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"kube-root-ca.crt\""
Sep 10 00:12:30.875477 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.875360    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"openshift-service-ca.crt\""
Sep 10 00:12:30.876291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.876277    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"default-dockercfg-bkdjc\""
Sep 10 00:12:30.876360 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.876340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"auth-test-model-1de9a7-kserve-self-signed-certs\""
Sep 10 00:12:30.880285 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.880259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:12:30.941675 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.941675 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.941913 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjd8c\" (UniqueName: \"kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.941913 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.941913 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:30.942048 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:30.941959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.043593 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.043794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.043794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.043794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.043794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjd8c\" (UniqueName: \"kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.044009 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.043816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.044118 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.044095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.044202 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.044130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.044202 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.044144    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.046146 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.046121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.046265 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.046249    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.051556 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.051512    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjd8c\" (UniqueName: \"kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c\") pod \"auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.183919 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.183849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:31.457415 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:31.457389    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:12:31.458960 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:12:31.458935    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b79226d_703d_4e60_a11d_d475794018f9.slice/crio-c5b9b29a904c87a266e1ce6dbd728730bc49a735752ee365567c68d891ee25b3 WatchSource:0}: Error finding container c5b9b29a904c87a266e1ce6dbd728730bc49a735752ee365567c68d891ee25b3: Status 404 returned error can't find the container with id c5b9b29a904c87a266e1ce6dbd728730bc49a735752ee365567c68d891ee25b3
Sep 10 00:12:32.250414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:32.250379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" event={"ID":"8b79226d-703d-4e60-a11d-d475794018f9","Type":"ContainerStarted","Data":"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"}
Sep 10 00:12:32.250414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:32.250424    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" event={"ID":"8b79226d-703d-4e60-a11d-d475794018f9","Type":"ContainerStarted","Data":"c5b9b29a904c87a266e1ce6dbd728730bc49a735752ee365567c68d891ee25b3"}
Sep 10 00:12:32.267674 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:32.267627    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" podStartSLOduration=2.267612378 podStartE2EDuration="2.267612378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:12:32.264944362 +0000 UTC m=+1425.585320426" watchObservedRunningTime="2026-09-10 00:12:32.267612378 +0000 UTC m=+1425.587988442"
Sep 10 00:12:41.184717 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:41.184630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:41.184717 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:41.184689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:41.203314 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:41.203288    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:12:41.313271 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:12:41.313239    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:13:51.480372 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.480335    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:13:51.480984 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.480597    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" podUID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" containerName="maas-api" containerID="cri-o://868210aea81356090be2e47922139f9363369448d505e536edac264f0a00c7cd" gracePeriod=30
Sep 10 00:13:51.534650 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.534613    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:13:51.534859 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.534832    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" podUID="7c7cba04-6d05-44f9-9079-3674eba53686" containerName="payload-processing" containerID="cri-o://840d2f03866e4ec03b20c91574ca89ae6273e068b933a522d7e343b0417ebcb1" gracePeriod=30
Sep 10 00:13:51.721649 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.721616    2576 generic.go:356] "Generic (PLEG): container finished" podID="7c7cba04-6d05-44f9-9079-3674eba53686" containerID="840d2f03866e4ec03b20c91574ca89ae6273e068b933a522d7e343b0417ebcb1" exitCode=0
Sep 10 00:13:51.721817 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.721713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" event={"ID":"7c7cba04-6d05-44f9-9079-3674eba53686","Type":"ContainerDied","Data":"840d2f03866e4ec03b20c91574ca89ae6273e068b933a522d7e343b0417ebcb1"}
Sep 10 00:13:51.723715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.723685    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" containerID="868210aea81356090be2e47922139f9363369448d505e536edac264f0a00c7cd" exitCode=0
Sep 10 00:13:51.723835 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.723788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" event={"ID":"cc745fac-caf3-49f1-9ccb-559b99f5ef89","Type":"ContainerDied","Data":"868210aea81356090be2e47922139f9363369448d505e536edac264f0a00c7cd"}
Sep 10 00:13:51.791024 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.791002    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:13:51.825142 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.825113    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls\") pod \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") "
Sep 10 00:13:51.825283 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.825161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca\") pod \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") "
Sep 10 00:13:51.825283 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.825254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs\") pod \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") "
Sep 10 00:13:51.825669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.825640    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc745fac-caf3-49f1-9ccb-559b99f5ef89" (UID: "cc745fac-caf3-49f1-9ccb-559b99f5ef89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:13:51.825842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.825800    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwl6g\" (UniqueName: \"kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g\") pod \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\" (UID: \"cc745fac-caf3-49f1-9ccb-559b99f5ef89\") "
Sep 10 00:13:51.826198 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.826176    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc745fac-caf3-49f1-9ccb-559b99f5ef89-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:51.828024 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.827995    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc745fac-caf3-49f1-9ccb-559b99f5ef89" (UID: "cc745fac-caf3-49f1-9ccb-559b99f5ef89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:13:51.828131 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.828038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc745fac-caf3-49f1-9ccb-559b99f5ef89" (UID: "cc745fac-caf3-49f1-9ccb-559b99f5ef89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:13:51.828279 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.828256    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g" (OuterVolumeSpecName: "kube-api-access-gwl6g") pod "cc745fac-caf3-49f1-9ccb-559b99f5ef89" (UID: "cc745fac-caf3-49f1-9ccb-559b99f5ef89"). InnerVolumeSpecName "kube-api-access-gwl6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:13:51.834715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.834694    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:13:51.927233 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b68rd\" (UniqueName: \"kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd\") pod \"7c7cba04-6d05-44f9-9079-3674eba53686\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") "
Sep 10 00:13:51.927430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927286    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume\") pod \"7c7cba04-6d05-44f9-9079-3674eba53686\" (UID: \"7c7cba04-6d05-44f9-9079-3674eba53686\") "
Sep 10 00:13:51.927564 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927520    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl6g\" (UniqueName: \"kubernetes.io/projected/cc745fac-caf3-49f1-9ccb-559b99f5ef89-kube-api-access-gwl6g\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:51.927679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927571    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:51.927679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927586    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc745fac-caf3-49f1-9ccb-559b99f5ef89-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:51.927769 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.927687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c7cba04-6d05-44f9-9079-3674eba53686" (UID: "7c7cba04-6d05-44f9-9079-3674eba53686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:13:51.929584 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:51.929559    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd" (OuterVolumeSpecName: "kube-api-access-b68rd") pod "7c7cba04-6d05-44f9-9079-3674eba53686" (UID: "7c7cba04-6d05-44f9-9079-3674eba53686"). InnerVolumeSpecName "kube-api-access-b68rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:13:52.029017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.028938    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68rd\" (UniqueName: \"kubernetes.io/projected/7c7cba04-6d05-44f9-9079-3674eba53686-kube-api-access-b68rd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:52.029017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.028965    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7cba04-6d05-44f9-9079-3674eba53686-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:13:52.731225 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.731193    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"
Sep 10 00:13:52.731225 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.731200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv" event={"ID":"7c7cba04-6d05-44f9-9079-3674eba53686","Type":"ContainerDied","Data":"ae2aaf3e674e3b45e8cede4f53d6a44a51eb58f91c3a65b8561c3ed6b6671b65"}
Sep 10 00:13:52.731751 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.731267    2576 scope.go:117] "RemoveContainer" containerID="840d2f03866e4ec03b20c91574ca89ae6273e068b933a522d7e343b0417ebcb1"
Sep 10 00:13:52.732890 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.732854    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"
Sep 10 00:13:52.732890 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.732871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr" event={"ID":"cc745fac-caf3-49f1-9ccb-559b99f5ef89","Type":"ContainerDied","Data":"22b730414ce3f592a1d9a9d57dec48740c90f85590f20432949c6bb93539dbc5"}
Sep 10 00:13:52.750910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.750891    2576 scope.go:117] "RemoveContainer" containerID="868210aea81356090be2e47922139f9363369448d505e536edac264f0a00c7cd"
Sep 10 00:13:52.764429 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.764409    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:13:52.771942 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.771921    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-557790f7-6f57686bbf-p8xhv"]
Sep 10 00:13:52.783917 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.783893    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:13:52.788789 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:52.788768    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-557790f7-b7dc59f7-q9bbr"]
Sep 10 00:13:53.203018 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:53.202986    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c7cba04-6d05-44f9-9079-3674eba53686" path="/var/lib/kubelet/pods/7c7cba04-6d05-44f9-9079-3674eba53686/volumes"
Sep 10 00:13:53.203383 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:53.203369    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" path="/var/lib/kubelet/pods/cc745fac-caf3-49f1-9ccb-559b99f5ef89/volumes"
Sep 10 00:13:59.830615 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:13:59.830577    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:14:00.448308 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448275    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:14:00.448696 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448682    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c7cba04-6d05-44f9-9079-3674eba53686" containerName="payload-processing"
Sep 10 00:14:00.448748 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448699    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c7cba04-6d05-44f9-9079-3674eba53686" containerName="payload-processing"
Sep 10 00:14:00.448748 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448718    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" containerName="maas-api"
Sep 10 00:14:00.448748 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448723    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" containerName="maas-api"
Sep 10 00:14:00.448843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448782    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c7cba04-6d05-44f9-9079-3674eba53686" containerName="payload-processing"
Sep 10 00:14:00.448843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.448795    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc745fac-caf3-49f1-9ccb-559b99f5ef89" containerName="maas-api"
Sep 10 00:14:00.453517 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.453498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.457207 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.457180    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"openshift-service-ca.crt\""
Sep 10 00:14:00.457207 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.457202    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"default-dockercfg-jxkgn\""
Sep 10 00:14:00.457392 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.457225    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"auth-test-model-b1baac-kserve-self-signed-certs\""
Sep 10 00:14:00.457392 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.457180    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"kube-root-ca.crt\""
Sep 10 00:14:00.462224 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.462204    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:14:00.509846 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.509821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.509988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.509850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.509988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.509942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.509988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.509975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.510147 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.509998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.510147 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.510099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzvb\" (UniqueName: \"kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.610860 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.610860 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611049 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611049 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611049 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611049 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.610980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzvb\" (UniqueName: \"kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611301 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.611277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611363 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.611305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.611402 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.611363    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.613407 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.613387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.613550 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.613518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.619486 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.619434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzvb\" (UniqueName: \"kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb\") pod \"auth-test-model-b1baac-kserve-f8486db95-j4qk6\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:00.763781 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:00.763714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:01.040375 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:01.040335    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:14:01.041240 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:14:01.041206    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7938db7_5837_4808_9450_45a07ba8f258.slice/crio-5004332664cb81296bd91a12a4db262c3b989f8c6737329056cfc1d9627a3f8f WatchSource:0}: Error finding container 5004332664cb81296bd91a12a4db262c3b989f8c6737329056cfc1d9627a3f8f: Status 404 returned error can't find the container with id 5004332664cb81296bd91a12a4db262c3b989f8c6737329056cfc1d9627a3f8f
Sep 10 00:14:01.789627 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:01.789591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" event={"ID":"e7938db7-5837-4808-9450-45a07ba8f258","Type":"ContainerStarted","Data":"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"}
Sep 10 00:14:01.789814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:01.789637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" event={"ID":"e7938db7-5837-4808-9450-45a07ba8f258","Type":"ContainerStarted","Data":"5004332664cb81296bd91a12a4db262c3b989f8c6737329056cfc1d9627a3f8f"}
Sep 10 00:14:01.807433 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:01.807388    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" podStartSLOduration=1.807373414 podStartE2EDuration="1.807373414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:14:01.806747155 +0000 UTC m=+1515.127123219" watchObservedRunningTime="2026-09-10 00:14:01.807373414 +0000 UTC m=+1515.127749477"
Sep 10 00:14:10.764304 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:10.764213    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:10.764304 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:10.764271    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:10.776851 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:10.776827    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:10.853462 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:10.853436    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:14:14.850818 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:14.850772    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" podUID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" containerName="main" containerID="cri-o://a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa" gracePeriod=45
Sep 10 00:14:15.091005 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.090979    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:14:15.158187 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158114    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158187 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158170    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh6vh\" (UniqueName: \"kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158196    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158329    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm\") pod \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\" (UID: \"fb4568c3-2cee-4c1d-8dda-b95c474081b5\") "
Sep 10 00:14:15.158679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:14:15.158679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158567    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home" (OuterVolumeSpecName: "home") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:14:15.158679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.158660    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:14:15.159098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.159076    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.159098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.159097    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.159284 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.159125    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.160764 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.160744    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm" (OuterVolumeSpecName: "dshm") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:14:15.161062 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.161044    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh" (OuterVolumeSpecName: "kube-api-access-lh6vh") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "kube-api-access-lh6vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:14:15.161136 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.161069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb4568c3-2cee-4c1d-8dda-b95c474081b5" (UID: "fb4568c3-2cee-4c1d-8dda-b95c474081b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:14:15.259871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.259845    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb4568c3-2cee-4c1d-8dda-b95c474081b5-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.259871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.259868    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb4568c3-2cee-4c1d-8dda-b95c474081b5-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.260026 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.259877    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6vh\" (UniqueName: \"kubernetes.io/projected/fb4568c3-2cee-4c1d-8dda-b95c474081b5-kube-api-access-lh6vh\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:14:15.871828 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.871788    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" containerID="a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa" exitCode=0
Sep 10 00:14:15.872228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.871865    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"
Sep 10 00:14:15.872228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.871876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" event={"ID":"fb4568c3-2cee-4c1d-8dda-b95c474081b5","Type":"ContainerDied","Data":"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"}
Sep 10 00:14:15.872228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.871925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq" event={"ID":"fb4568c3-2cee-4c1d-8dda-b95c474081b5","Type":"ContainerDied","Data":"481aee7b9a9aa9f103ec5bb97a9139be7cbf4e526ee6e3d0046609a38c8ff406"}
Sep 10 00:14:15.872228 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.871943    2576 scope.go:117] "RemoveContainer" containerID="a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"
Sep 10 00:14:15.881178 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.881159    2576 scope.go:117] "RemoveContainer" containerID="a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"
Sep 10 00:14:15.881473 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:14:15.881455    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa\": container with ID starting with a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa not found: ID does not exist" containerID="a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"
Sep 10 00:14:15.881524 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.881482    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa"} err="failed to get container status \"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa\": rpc error: code = NotFound desc = could not find container \"a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa\": container with ID starting with a90a468bdecf253046dd6502cd275a5b2190ee62c071953670b5765e614ee2fa not found: ID does not exist"
Sep 10 00:14:15.889605 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.889581    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:14:15.892991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:15.892971    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-557790f7/e2e-lifecycle-model-557790f7-kserve-6f85c5d999-m86zq"]
Sep 10 00:14:17.205122 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:14:17.205086    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" path="/var/lib/kubelet/pods/fb4568c3-2cee-4c1d-8dda-b95c474081b5/volumes"
Sep 10 00:15:00.148466 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.148432    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"]
Sep 10 00:15:00.148932 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.148914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" containerName="main"
Sep 10 00:15:00.148932 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.148929    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" containerName="main"
Sep 10 00:15:00.149008 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.148992    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4568c3-2cee-4c1d-8dda-b95c474081b5" containerName="main"
Sep 10 00:15:00.153688 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.153668    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"
Sep 10 00:15:00.156527 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.156505    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:15:00.164111 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.163410    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"]
Sep 10 00:15:00.167650 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.167628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"
Sep 10 00:15:00.446206 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:00.446182    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"]
Sep 10 00:15:00.448395 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:15:00.448366    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a326325_c11f_42bb_a290_1f48e7a33260.slice/crio-1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce WatchSource:0}: Error finding container 1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce: Status 404 returned error can't find the container with id 1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce
Sep 10 00:15:01.138086 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:01.138055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv" event={"ID":"5a326325-c11f-42bb-a290-1f48e7a33260","Type":"ContainerStarted","Data":"1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce"}
Sep 10 00:15:04.156640 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:04.156608    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a326325-c11f-42bb-a290-1f48e7a33260" containerID="1896156476a78b2027114113c1cbbaff81f92f5f0cfb3cce4cc1668fe2d9e86c" exitCode=0
Sep 10 00:15:04.157032 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:04.156694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv" event={"ID":"5a326325-c11f-42bb-a290-1f48e7a33260","Type":"ContainerDied","Data":"1896156476a78b2027114113c1cbbaff81f92f5f0cfb3cce4cc1668fe2d9e86c"}
Sep 10 00:15:05.302193 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:05.302167    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"
Sep 10 00:15:06.169211 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:06.169178    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"
Sep 10 00:15:06.169399 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:06.169226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv" event={"ID":"5a326325-c11f-42bb-a290-1f48e7a33260","Type":"ContainerDied","Data":"1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce"}
Sep 10 00:15:06.169399 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:15:06.169269    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b058f8c67c8201bb0c86f3f344153784519ed30944d734f82308cf7033d52ce"
Sep 10 00:16:13.112253 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.112212    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:16:13.112861 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.112838    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a326325-c11f-42bb-a290-1f48e7a33260" containerName="cleanup"
Sep 10 00:16:13.112947 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.112864    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a326325-c11f-42bb-a290-1f48e7a33260" containerName="cleanup"
Sep 10 00:16:13.113017 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.113004    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a326325-c11f-42bb-a290-1f48e7a33260" containerName="cleanup"
Sep 10 00:16:13.116379 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.116356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.118856 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.118832    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-beac5e-serving-cert\""
Sep 10 00:16:13.129042 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.129015    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:16:13.199556 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.199509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.199711 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.199668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.199764 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.199713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.199764 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.199749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fc29\" (UniqueName: \"kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.300614 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.300581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.300804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.300631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.300804 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.300716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fc29\" (UniqueName: \"kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.300936 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.300815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.301144 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:13.301119    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-serving-cert: secret "maas-api-e2e-api-a-beac5e-serving-cert" not found
Sep 10 00:16:13.301250 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:13.301206    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls podName:b1947068-fe5a-46a3-827e-a0c4318a86c3 nodeName:}" failed. No retries permitted until 2026-09-10 00:16:13.801184581 +0000 UTC m=+1647.121560626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls") pod "maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" (UID: "b1947068-fe5a-46a3-827e-a0c4318a86c3") : secret "maas-api-e2e-api-a-beac5e-serving-cert" not found
Sep 10 00:16:13.301501 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.301481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.303629 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.303605    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.312238 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.312218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fc29\" (UniqueName: \"kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.805256 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.805211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.807842 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.807820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") pod \"maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:13.814020 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.813994    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:16:13.818462 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.818429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:13.822098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.822036    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-beac5e\""
Sep 10 00:16:13.822214 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.822095    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-beac5e-dockercfg-db6hg\""
Sep 10 00:16:13.825888 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.825867    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:16:13.906434 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.906400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:13.906638 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:13.906581    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hck\" (UniqueName: \"kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.007303 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.007268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.007481 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.007394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hck\" (UniqueName: \"kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.008119 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.008097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.015902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.015881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hck\" (UniqueName: \"kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck\") pod \"payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.027704 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.027683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:14.130429 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.130402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.301087 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.301057    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:16:14.301287 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:16:14.301265    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1947068_fe5a_46a3_827e_a0c4318a86c3.slice/crio-b525163dd5734ded931f753b47bc2f621b85b5a08a7db412addccf85ef929358 WatchSource:0}: Error finding container b525163dd5734ded931f753b47bc2f621b85b5a08a7db412addccf85ef929358: Status 404 returned error can't find the container with id b525163dd5734ded931f753b47bc2f621b85b5a08a7db412addccf85ef929358
Sep 10 00:16:14.302721 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.302704    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:16:14.409822 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.409792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:16:14.410920 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:16:14.410890    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5446661_0def_44a9_8219_b352090db0c3.slice/crio-abd67c8ef849c3f8d5cd6d8307ca9ab864d8499efa670edab038c4f85ccbc7db WatchSource:0}: Error finding container abd67c8ef849c3f8d5cd6d8307ca9ab864d8499efa670edab038c4f85ccbc7db: Status 404 returned error can't find the container with id abd67c8ef849c3f8d5cd6d8307ca9ab864d8499efa670edab038c4f85ccbc7db
Sep 10 00:16:14.558610 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.558577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" event={"ID":"d5446661-0def-44a9-8219-b352090db0c3","Type":"ContainerStarted","Data":"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"}
Sep 10 00:16:14.558797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.558642    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" event={"ID":"d5446661-0def-44a9-8219-b352090db0c3","Type":"ContainerStarted","Data":"abd67c8ef849c3f8d5cd6d8307ca9ab864d8499efa670edab038c4f85ccbc7db"}
Sep 10 00:16:14.558797 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.558767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:14.559954 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.559934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" event={"ID":"b1947068-fe5a-46a3-827e-a0c4318a86c3","Type":"ContainerStarted","Data":"b525163dd5734ded931f753b47bc2f621b85b5a08a7db412addccf85ef929358"}
Sep 10 00:16:14.587474 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:14.587417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" podStartSLOduration=1.587401531 podStartE2EDuration="1.587401531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:14.574641867 +0000 UTC m=+1647.895017956" watchObservedRunningTime="2026-09-10 00:16:14.587401531 +0000 UTC m=+1647.907777594"
Sep 10 00:16:15.567043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:15.567000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" event={"ID":"b1947068-fe5a-46a3-827e-a0c4318a86c3","Type":"ContainerStarted","Data":"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"}
Sep 10 00:16:15.567508 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:15.567071    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:15.584960 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:15.584911    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" podStartSLOduration=2.159402191 podStartE2EDuration="2.584894692s" totalImagesPullingTime="425.492501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:15.581968886 +0000 UTC m=+1648.902344986" watchObservedRunningTime="2026-09-10 00:16:15.584894692 +0000 UTC m=+1648.905270785"
Sep 10 00:16:21.579888 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:21.579859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:16:25.569651 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:25.569617    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:16:42.153114 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.153083    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:16:42.157153 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.157127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.159678 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.159650    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b9483e-serving-cert\""
Sep 10 00:16:42.169679 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.169656    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:16:42.290940 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.290877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.291132 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.290947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.291132 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.290994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46cc\" (UniqueName: \"kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.291132 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.291029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.391800 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.391768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.391992 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.391810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.391992 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.391851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46cc\" (UniqueName: \"kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.391992 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:42.391937    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-serving-cert: secret "maas-api-e2e-api-b-b9483e-serving-cert" not found
Sep 10 00:16:42.391992 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.391952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.392208 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:42.392014    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls podName:ebe7a4ae-1feb-41e1-9235-dd9a9825359d nodeName:}" failed. No retries permitted until 2026-09-10 00:16:42.891991476 +0000 UTC m=+1676.212367523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls") pod "maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d") : secret "maas-api-e2e-api-b-b9483e-serving-cert" not found
Sep 10 00:16:42.392588 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.392568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.394744 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.394726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.399930 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.399905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46cc\" (UniqueName: \"kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.863097 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.863062    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:16:42.866766 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.866748    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:42.869225 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.869204    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b9483e\""
Sep 10 00:16:42.869346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.869207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b9483e-dockercfg-c922k\""
Sep 10 00:16:42.875107 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.875087    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:16:42.896291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.896263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:42.896419 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:42.896403    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-serving-cert: secret "maas-api-e2e-api-b-b9483e-serving-cert" not found
Sep 10 00:16:42.896482 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:16:42.896472    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls podName:ebe7a4ae-1feb-41e1-9235-dd9a9825359d nodeName:}" failed. No retries permitted until 2026-09-10 00:16:43.896455211 +0000 UTC m=+1677.216831261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls") pod "maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d") : secret "maas-api-e2e-api-b-b9483e-serving-cert" not found
Sep 10 00:16:42.902498 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.902468    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:16:42.906749 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.906734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:42.915180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.915156    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:16:42.997867 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.997838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxnr\" (UniqueName: \"kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:42.998034 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:42.997928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.098486 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.098442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.098695 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.098530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.098695 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.098601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95qn\" (UniqueName: \"kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.098816 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.098694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxnr\" (UniqueName: \"kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.099141 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.099121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.106414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.106384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxnr\" (UniqueName: \"kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr\") pod \"payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.181403 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.181317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.199516 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.199486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.199922 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.199569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95qn\" (UniqueName: \"kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.200687 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.200660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.209982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.209956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95qn\" (UniqueName: \"kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn\") pod \"payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.218551 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.218513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.457899 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.457872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:16:43.459096 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:16:43.459071    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df8474d_3c76_4c31_ad84_3489fd173307.slice/crio-e5818e728ea179c7fe7627f9767a63183dd7a5cabcc49ef7bcbc770f3ac5eaa7 WatchSource:0}: Error finding container e5818e728ea179c7fe7627f9767a63183dd7a5cabcc49ef7bcbc770f3ac5eaa7: Status 404 returned error can't find the container with id e5818e728ea179c7fe7627f9767a63183dd7a5cabcc49ef7bcbc770f3ac5eaa7
Sep 10 00:16:43.508437 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.508414    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:16:43.508805 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:16:43.508783    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80617e9_27a2_4395_b560_2255a521d04d.slice/crio-e8d947f7aa437a0d7a4e5bd4203bb9129a781bd66c1193b12bcf5473ec6e4598 WatchSource:0}: Error finding container e8d947f7aa437a0d7a4e5bd4203bb9129a781bd66c1193b12bcf5473ec6e4598: Status 404 returned error can't find the container with id e8d947f7aa437a0d7a4e5bd4203bb9129a781bd66c1193b12bcf5473ec6e4598
Sep 10 00:16:43.741001 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.740912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" event={"ID":"e80617e9-27a2-4395-b560-2255a521d04d","Type":"ContainerStarted","Data":"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"}
Sep 10 00:16:43.741001 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.740975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" event={"ID":"e80617e9-27a2-4395-b560-2255a521d04d","Type":"ContainerStarted","Data":"e8d947f7aa437a0d7a4e5bd4203bb9129a781bd66c1193b12bcf5473ec6e4598"}
Sep 10 00:16:43.741193 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.741005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:43.742635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.742604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" event={"ID":"9df8474d-3c76-4c31-ad84-3489fd173307","Type":"ContainerStarted","Data":"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"}
Sep 10 00:16:43.742755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.742655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" event={"ID":"9df8474d-3c76-4c31-ad84-3489fd173307","Type":"ContainerStarted","Data":"e5818e728ea179c7fe7627f9767a63183dd7a5cabcc49ef7bcbc770f3ac5eaa7"}
Sep 10 00:16:43.742755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.742705    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:16:43.760370 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.760329    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" podStartSLOduration=1.7603156009999998 podStartE2EDuration="1.760315601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:43.756573746 +0000 UTC m=+1677.076949807" watchObservedRunningTime="2026-09-10 00:16:43.760315601 +0000 UTC m=+1677.080692005"
Sep 10 00:16:43.773714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.773674    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" podStartSLOduration=1.773660958 podStartE2EDuration="1.773660958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:43.770956549 +0000 UTC m=+1677.091332607" watchObservedRunningTime="2026-09-10 00:16:43.773660958 +0000 UTC m=+1677.094037021"
Sep 10 00:16:43.907798 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.907767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:43.910525 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.910495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:43.971593 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:43.971562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:44.250835 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:44.250805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:16:44.251493 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:16:44.251465    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe7a4ae_1feb_41e1_9235_dd9a9825359d.slice/crio-f174edcf6e8173bcf2516208f3983cfd4ed4707719f1393d4377b9ad043ada34 WatchSource:0}: Error finding container f174edcf6e8173bcf2516208f3983cfd4ed4707719f1393d4377b9ad043ada34: Status 404 returned error can't find the container with id f174edcf6e8173bcf2516208f3983cfd4ed4707719f1393d4377b9ad043ada34
Sep 10 00:16:44.751392 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:44.751325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" event={"ID":"ebe7a4ae-1feb-41e1-9235-dd9a9825359d","Type":"ContainerStarted","Data":"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"}
Sep 10 00:16:44.751560 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:44.751398    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:44.751560 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:44.751425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" event={"ID":"ebe7a4ae-1feb-41e1-9235-dd9a9825359d","Type":"ContainerStarted","Data":"f174edcf6e8173bcf2516208f3983cfd4ed4707719f1393d4377b9ad043ada34"}
Sep 10 00:16:44.771457 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:44.771403    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" podStartSLOduration=2.366555896 podStartE2EDuration="2.771385555s" totalImagesPullingTime="404.829659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:44.766995329 +0000 UTC m=+1678.087371396" watchObservedRunningTime="2026-09-10 00:16:44.771385555 +0000 UTC m=+1678.091761620"
Sep 10 00:16:50.763586 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:50.763560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:16:54.754104 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:54.754075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:16:54.754713 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:16:54.754691    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:17:24.905813 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:24.905709    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:17:24.906450 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:24.905958    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" podUID="b1947068-fe5a-46a3-827e-a0c4318a86c3" containerName="maas-api" containerID="cri-o://c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593" gracePeriod=30
Sep 10 00:17:24.952324 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:24.952292    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:17:24.952530 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:24.952504    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" podUID="d5446661-0def-44a9-8219-b352090db0c3" containerName="payload-pre-processing" containerID="cri-o://f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98" gracePeriod=30
Sep 10 00:17:25.213201 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.213169    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:17:25.247612 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.247589    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:17:25.310119 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310089    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66hck\" (UniqueName: \"kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck\") pod \"d5446661-0def-44a9-8219-b352090db0c3\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") "
Sep 10 00:17:25.310119 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca\") pod \"b1947068-fe5a-46a3-827e-a0c4318a86c3\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") "
Sep 10 00:17:25.310333 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310148    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fc29\" (UniqueName: \"kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29\") pod \"b1947068-fe5a-46a3-827e-a0c4318a86c3\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") "
Sep 10 00:17:25.310333 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs\") pod \"b1947068-fe5a-46a3-827e-a0c4318a86c3\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") "
Sep 10 00:17:25.310420 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310401    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume\") pod \"d5446661-0def-44a9-8219-b352090db0c3\" (UID: \"d5446661-0def-44a9-8219-b352090db0c3\") "
Sep 10 00:17:25.310482 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") pod \"b1947068-fe5a-46a3-827e-a0c4318a86c3\" (UID: \"b1947068-fe5a-46a3-827e-a0c4318a86c3\") "
Sep 10 00:17:25.310591 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310496    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1947068-fe5a-46a3-827e-a0c4318a86c3" (UID: "b1947068-fe5a-46a3-827e-a0c4318a86c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:25.310846 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310807    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5446661-0def-44a9-8219-b352090db0c3" (UID: "d5446661-0def-44a9-8219-b352090db0c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:25.310945 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.310863    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1947068-fe5a-46a3-827e-a0c4318a86c3-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.312455 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.312431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1947068-fe5a-46a3-827e-a0c4318a86c3" (UID: "b1947068-fe5a-46a3-827e-a0c4318a86c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:17:25.312782 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.312764    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1947068-fe5a-46a3-827e-a0c4318a86c3" (UID: "b1947068-fe5a-46a3-827e-a0c4318a86c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:17:25.313036 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.313014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck" (OuterVolumeSpecName: "kube-api-access-66hck") pod "d5446661-0def-44a9-8219-b352090db0c3" (UID: "d5446661-0def-44a9-8219-b352090db0c3"). InnerVolumeSpecName "kube-api-access-66hck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:25.313036 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.313022    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29" (OuterVolumeSpecName: "kube-api-access-8fc29") pod "b1947068-fe5a-46a3-827e-a0c4318a86c3" (UID: "b1947068-fe5a-46a3-827e-a0c4318a86c3"). InnerVolumeSpecName "kube-api-access-8fc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:25.411776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.411700    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hck\" (UniqueName: \"kubernetes.io/projected/d5446661-0def-44a9-8219-b352090db0c3-kube-api-access-66hck\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.411776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.411727    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fc29\" (UniqueName: \"kubernetes.io/projected/b1947068-fe5a-46a3-827e-a0c4318a86c3-kube-api-access-8fc29\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.411776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.411738    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.411776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.411746    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5446661-0def-44a9-8219-b352090db0c3-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.411776 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.411755    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1947068-fe5a-46a3-827e-a0c4318a86c3-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.995169 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.995133    2576 generic.go:356] "Generic (PLEG): container finished" podID="b1947068-fe5a-46a3-827e-a0c4318a86c3" containerID="c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593" exitCode=0
Sep 10 00:17:25.995640 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.995199    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"
Sep 10 00:17:25.995640 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.995224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" event={"ID":"b1947068-fe5a-46a3-827e-a0c4318a86c3","Type":"ContainerDied","Data":"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"}
Sep 10 00:17:25.995640 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.995306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62" event={"ID":"b1947068-fe5a-46a3-827e-a0c4318a86c3","Type":"ContainerDied","Data":"b525163dd5734ded931f753b47bc2f621b85b5a08a7db412addccf85ef929358"}
Sep 10 00:17:25.995640 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.995329    2576 scope.go:117] "RemoveContainer" containerID="c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"
Sep 10 00:17:25.996782 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.996764    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5446661-0def-44a9-8219-b352090db0c3" containerID="f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98" exitCode=0
Sep 10 00:17:25.996865 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.996796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" event={"ID":"d5446661-0def-44a9-8219-b352090db0c3","Type":"ContainerDied","Data":"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"}
Sep 10 00:17:25.996865 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.996832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h" event={"ID":"d5446661-0def-44a9-8219-b352090db0c3","Type":"ContainerDied","Data":"abd67c8ef849c3f8d5cd6d8307ca9ab864d8499efa670edab038c4f85ccbc7db"}
Sep 10 00:17:25.996982 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:25.996895    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"
Sep 10 00:17:26.008065 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.008036    2576 scope.go:117] "RemoveContainer" containerID="c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"
Sep 10 00:17:26.008379 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:17:26.008358    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593\": container with ID starting with c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593 not found: ID does not exist" containerID="c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"
Sep 10 00:17:26.008442 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.008388    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593"} err="failed to get container status \"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593\": rpc error: code = NotFound desc = could not find container \"c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593\": container with ID starting with c00dbb053ba0d7b4f6db62c3b13271b5ced05eb2958cc1d3e60c1db132c0c593 not found: ID does not exist"
Sep 10 00:17:26.008442 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.008404    2576 scope.go:117] "RemoveContainer" containerID="f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"
Sep 10 00:17:26.018221 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.018200    2576 scope.go:117] "RemoveContainer" containerID="f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"
Sep 10 00:17:26.018769 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:17:26.018719    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98\": container with ID starting with f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98 not found: ID does not exist" containerID="f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"
Sep 10 00:17:26.018880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.018781    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98"} err="failed to get container status \"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98\": rpc error: code = NotFound desc = could not find container \"f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98\": container with ID starting with f527b8aed7c8f088565d0dedd878e82b960a7c16be2d7bcbc7362db047d55f98 not found: ID does not exist"
Sep 10 00:17:26.021774 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.021708    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:17:26.024855 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.024833    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-beac5e-bcbbcf8c8-dwt9h"]
Sep 10 00:17:26.033878 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.033852    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:17:26.037347 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:26.037324    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-beac5e-6bb9cdcdb7-6fw62"]
Sep 10 00:17:27.202889 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:27.202855    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1947068-fe5a-46a3-827e-a0c4318a86c3" path="/var/lib/kubelet/pods/b1947068-fe5a-46a3-827e-a0c4318a86c3/volumes"
Sep 10 00:17:27.203433 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:27.203412    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5446661-0def-44a9-8219-b352090db0c3" path="/var/lib/kubelet/pods/d5446661-0def-44a9-8219-b352090db0c3/volumes"
Sep 10 00:17:47.306987 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.306954    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:17:47.307596 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.307196    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" podUID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" containerName="maas-api" containerID="cri-o://0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f" gracePeriod=30
Sep 10 00:17:47.355845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.355813    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:17:47.356090 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.356050    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" podUID="e80617e9-27a2-4395-b560-2255a521d04d" containerName="payload-processing" containerID="cri-o://415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff" gracePeriod=30
Sep 10 00:17:47.363818 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.363793    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:17:47.364029 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.364007    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" podUID="9df8474d-3c76-4c31-ad84-3489fd173307" containerName="payload-pre-processing" containerID="cri-o://dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8" gracePeriod=30
Sep 10 00:17:47.622861 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.622835    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:17:47.659715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.659691    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:17:47.688836 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.688813    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:17:47.725655 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725630    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume\") pod \"e80617e9-27a2-4395-b560-2255a521d04d\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") "
Sep 10 00:17:47.725811 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725726    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h46cc\" (UniqueName: \"kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc\") pod \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") "
Sep 10 00:17:47.725811 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f95qn\" (UniqueName: \"kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn\") pod \"e80617e9-27a2-4395-b560-2255a521d04d\" (UID: \"e80617e9-27a2-4395-b560-2255a521d04d\") "
Sep 10 00:17:47.725811 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725789    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") pod \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") "
Sep 10 00:17:47.725996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725854    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs\") pod \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") "
Sep 10 00:17:47.725996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.725889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca\") pod \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\" (UID: \"ebe7a4ae-1feb-41e1-9235-dd9a9825359d\") "
Sep 10 00:17:47.726523 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.726494    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebe7a4ae-1feb-41e1-9235-dd9a9825359d" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:47.726880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.726858    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e80617e9-27a2-4395-b560-2255a521d04d" (UID: "e80617e9-27a2-4395-b560-2255a521d04d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:47.728731 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.728702    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn" (OuterVolumeSpecName: "kube-api-access-f95qn") pod "e80617e9-27a2-4395-b560-2255a521d04d" (UID: "e80617e9-27a2-4395-b560-2255a521d04d"). InnerVolumeSpecName "kube-api-access-f95qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:47.728846 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.728737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc" (OuterVolumeSpecName: "kube-api-access-h46cc") pod "ebe7a4ae-1feb-41e1-9235-dd9a9825359d" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d"). InnerVolumeSpecName "kube-api-access-h46cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:47.728846 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.728748    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebe7a4ae-1feb-41e1-9235-dd9a9825359d" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:17:47.729069 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.729040    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebe7a4ae-1feb-41e1-9235-dd9a9825359d" (UID: "ebe7a4ae-1feb-41e1-9235-dd9a9825359d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:17:47.827419 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827392    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume\") pod \"9df8474d-3c76-4c31-ad84-3489fd173307\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") "
Sep 10 00:17:47.827613 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827503    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmxnr\" (UniqueName: \"kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr\") pod \"9df8474d-3c76-4c31-ad84-3489fd173307\" (UID: \"9df8474d-3c76-4c31-ad84-3489fd173307\") "
Sep 10 00:17:47.827805 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827789    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h46cc\" (UniqueName: \"kubernetes.io/projected/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-kube-api-access-h46cc\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827810    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f95qn\" (UniqueName: \"kubernetes.io/projected/e80617e9-27a2-4395-b560-2255a521d04d-kube-api-access-f95qn\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827820    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827830    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827841    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe7a4ae-1feb-41e1-9235-dd9a9825359d-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827857    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80617e9-27a2-4395-b560-2255a521d04d-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.827871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.827784    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9df8474d-3c76-4c31-ad84-3489fd173307" (UID: "9df8474d-3c76-4c31-ad84-3489fd173307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:47.829659 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.829636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr" (OuterVolumeSpecName: "kube-api-access-hmxnr") pod "9df8474d-3c76-4c31-ad84-3489fd173307" (UID: "9df8474d-3c76-4c31-ad84-3489fd173307"). InnerVolumeSpecName "kube-api-access-hmxnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:47.928949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.928915    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxnr\" (UniqueName: \"kubernetes.io/projected/9df8474d-3c76-4c31-ad84-3489fd173307-kube-api-access-hmxnr\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:47.928949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:47.928946    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df8474d-3c76-4c31-ad84-3489fd173307-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:17:48.144272 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.144181    2576 generic.go:356] "Generic (PLEG): container finished" podID="9df8474d-3c76-4c31-ad84-3489fd173307" containerID="dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8" exitCode=0
Sep 10 00:17:48.144272 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.144238    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"
Sep 10 00:17:48.144481 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.144266    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" event={"ID":"9df8474d-3c76-4c31-ad84-3489fd173307","Type":"ContainerDied","Data":"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"}
Sep 10 00:17:48.144481 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.144351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw" event={"ID":"9df8474d-3c76-4c31-ad84-3489fd173307","Type":"ContainerDied","Data":"e5818e728ea179c7fe7627f9767a63183dd7a5cabcc49ef7bcbc770f3ac5eaa7"}
Sep 10 00:17:48.144481 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.144380    2576 scope.go:117] "RemoveContainer" containerID="dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"
Sep 10 00:17:48.145872 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.145855    2576 generic.go:356] "Generic (PLEG): container finished" podID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" containerID="0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f" exitCode=0
Sep 10 00:17:48.145986 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.145928    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"
Sep 10 00:17:48.145986 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.145945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" event={"ID":"ebe7a4ae-1feb-41e1-9235-dd9a9825359d","Type":"ContainerDied","Data":"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"}
Sep 10 00:17:48.146094 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.145995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw" event={"ID":"ebe7a4ae-1feb-41e1-9235-dd9a9825359d","Type":"ContainerDied","Data":"f174edcf6e8173bcf2516208f3983cfd4ed4707719f1393d4377b9ad043ada34"}
Sep 10 00:17:48.147435 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.147414    2576 generic.go:356] "Generic (PLEG): container finished" podID="e80617e9-27a2-4395-b560-2255a521d04d" containerID="415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff" exitCode=0
Sep 10 00:17:48.147642 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.147459    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"
Sep 10 00:17:48.147642 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.147487    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" event={"ID":"e80617e9-27a2-4395-b560-2255a521d04d","Type":"ContainerDied","Data":"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"}
Sep 10 00:17:48.147642 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.147519    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq" event={"ID":"e80617e9-27a2-4395-b560-2255a521d04d","Type":"ContainerDied","Data":"e8d947f7aa437a0d7a4e5bd4203bb9129a781bd66c1193b12bcf5473ec6e4598"}
Sep 10 00:17:48.155257 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.155238    2576 scope.go:117] "RemoveContainer" containerID="dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"
Sep 10 00:17:48.155584 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:17:48.155558    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8\": container with ID starting with dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8 not found: ID does not exist" containerID="dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"
Sep 10 00:17:48.155669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.155591    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8"} err="failed to get container status \"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8\": rpc error: code = NotFound desc = could not find container \"dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8\": container with ID starting with dfcd2c948dff84fdb9e46fd723a20361ee57c6c5e9f199a1c41a53e42391fbe8 not found: ID does not exist"
Sep 10 00:17:48.155669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.155606    2576 scope.go:117] "RemoveContainer" containerID="0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"
Sep 10 00:17:48.164315 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.164299    2576 scope.go:117] "RemoveContainer" containerID="0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"
Sep 10 00:17:48.164621 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:17:48.164601    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f\": container with ID starting with 0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f not found: ID does not exist" containerID="0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"
Sep 10 00:17:48.164697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.164625    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f"} err="failed to get container status \"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f\": rpc error: code = NotFound desc = could not find container \"0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f\": container with ID starting with 0c4b54c7ddb70fb5225fa3493542f6dc08627972e5606ecf3e613daf57366d0f not found: ID does not exist"
Sep 10 00:17:48.164697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.164639    2576 scope.go:117] "RemoveContainer" containerID="415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"
Sep 10 00:17:48.173003 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.172984    2576 scope.go:117] "RemoveContainer" containerID="415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"
Sep 10 00:17:48.173272 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:17:48.173248    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff\": container with ID starting with 415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff not found: ID does not exist" containerID="415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"
Sep 10 00:17:48.173339 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.173282    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff"} err="failed to get container status \"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff\": rpc error: code = NotFound desc = could not find container \"415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff\": container with ID starting with 415b59b9a8357317179849a98f1a5a120c055c62b59e85604a48eb089307d7ff not found: ID does not exist"
Sep 10 00:17:48.175953 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.175934    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:17:48.181432 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.181410    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9483e-c574cc78-4tncw"]
Sep 10 00:17:48.191342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.191320    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:17:48.194821 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.194799    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b9483e-7c678dc876-bmdpw"]
Sep 10 00:17:48.205780 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.205750    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:17:48.209846 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:48.209828    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9483e-bfbb4cd95-8nztq"]
Sep 10 00:17:49.203129 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:49.203092    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df8474d-3c76-4c31-ad84-3489fd173307" path="/var/lib/kubelet/pods/9df8474d-3c76-4c31-ad84-3489fd173307/volumes"
Sep 10 00:17:49.203647 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:49.203525    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80617e9-27a2-4395-b560-2255a521d04d" path="/var/lib/kubelet/pods/e80617e9-27a2-4395-b560-2255a521d04d/volumes"
Sep 10 00:17:49.203941 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:17:49.203924    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" path="/var/lib/kubelet/pods/ebe7a4ae-1feb-41e1-9235-dd9a9825359d/volumes"
Sep 10 00:18:04.553195 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.553162    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:18:04.553690 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.553424    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" podUID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" containerName="payload-processing" containerID="cri-o://7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0" gracePeriod=30
Sep 10 00:18:04.578427 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.578384    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:18:04.578947 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.578916    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" podUID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" containerName="payload-pre-processing" containerID="cri-o://5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db" gracePeriod=30
Sep 10 00:18:04.839168 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.839141    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:18:04.876013 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.875983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:18:04.899259 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899232    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume\") pod \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") "
Sep 10 00:18:04.899669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxb72\" (UniqueName: \"kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72\") pod \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\" (UID: \"b1d3c2cc-a437-4135-a8e8-070eeabec58a\") "
Sep 10 00:18:04.899669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume\") pod \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") "
Sep 10 00:18:04.899669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899486    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4gxq\" (UniqueName: \"kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq\") pod \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\" (UID: \"05ebc238-edf1-4c4e-bf97-cd4551887c6b\") "
Sep 10 00:18:04.899962 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899925    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1d3c2cc-a437-4135-a8e8-070eeabec58a" (UID: "b1d3c2cc-a437-4135-a8e8-070eeabec58a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:04.900042 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.899952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05ebc238-edf1-4c4e-bf97-cd4551887c6b" (UID: "05ebc238-edf1-4c4e-bf97-cd4551887c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:04.901876 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.901848    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72" (OuterVolumeSpecName: "kube-api-access-vxb72") pod "b1d3c2cc-a437-4135-a8e8-070eeabec58a" (UID: "b1d3c2cc-a437-4135-a8e8-070eeabec58a"). InnerVolumeSpecName "kube-api-access-vxb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:04.901967 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:04.901939    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq" (OuterVolumeSpecName: "kube-api-access-t4gxq") pod "05ebc238-edf1-4c4e-bf97-cd4551887c6b" (UID: "05ebc238-edf1-4c4e-bf97-cd4551887c6b"). InnerVolumeSpecName "kube-api-access-t4gxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:05.000988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.000958    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gxq\" (UniqueName: \"kubernetes.io/projected/05ebc238-edf1-4c4e-bf97-cd4551887c6b-kube-api-access-t4gxq\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:05.000988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.000983    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1d3c2cc-a437-4135-a8e8-070eeabec58a-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:05.000988 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.000993    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxb72\" (UniqueName: \"kubernetes.io/projected/b1d3c2cc-a437-4135-a8e8-070eeabec58a-kube-api-access-vxb72\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:05.001186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.001002    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ebc238-edf1-4c4e-bf97-cd4551887c6b-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:05.249815 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.249782    2576 generic.go:356] "Generic (PLEG): container finished" podID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" containerID="5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db" exitCode=0
Sep 10 00:18:05.249948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.249821    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"
Sep 10 00:18:05.249948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.249866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" event={"ID":"b1d3c2cc-a437-4135-a8e8-070eeabec58a","Type":"ContainerDied","Data":"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"}
Sep 10 00:18:05.249948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.249914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6" event={"ID":"b1d3c2cc-a437-4135-a8e8-070eeabec58a","Type":"ContainerDied","Data":"e5d4f818293162fcdfe85bc35ec6a7c04904a1e2c3143602e670d4766469b796"}
Sep 10 00:18:05.249948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.249937    2576 scope.go:117] "RemoveContainer" containerID="5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"
Sep 10 00:18:05.251270 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.251248    2576 generic.go:356] "Generic (PLEG): container finished" podID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" containerID="7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0" exitCode=0
Sep 10 00:18:05.251397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.251281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" event={"ID":"05ebc238-edf1-4c4e-bf97-cd4551887c6b","Type":"ContainerDied","Data":"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"}
Sep 10 00:18:05.251397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.251302    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"
Sep 10 00:18:05.251397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.251335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg" event={"ID":"05ebc238-edf1-4c4e-bf97-cd4551887c6b","Type":"ContainerDied","Data":"69fe76f059f865cb88d093ea2c70352342a6c28226d9947234173714f34574f2"}
Sep 10 00:18:05.259614 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.259598    2576 scope.go:117] "RemoveContainer" containerID="5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"
Sep 10 00:18:05.259896 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:18:05.259876    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db\": container with ID starting with 5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db not found: ID does not exist" containerID="5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"
Sep 10 00:18:05.259939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.259905    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db"} err="failed to get container status \"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db\": rpc error: code = NotFound desc = could not find container \"5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db\": container with ID starting with 5688110c344cb85f9d0fa6a0402b6879b601a468519636f9985155cdbf0187db not found: ID does not exist"
Sep 10 00:18:05.259939 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.259921    2576 scope.go:117] "RemoveContainer" containerID="7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"
Sep 10 00:18:05.266278 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.266256    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:18:05.268617 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.268600    2576 scope.go:117] "RemoveContainer" containerID="7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"
Sep 10 00:18:05.269129 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:18:05.268997    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0\": container with ID starting with 7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0 not found: ID does not exist" containerID="7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"
Sep 10 00:18:05.269129 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.269032    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0"} err="failed to get container status \"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0\": rpc error: code = NotFound desc = could not find container \"7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0\": container with ID starting with 7a1c39c0c0ad09a8829ab4530350efd6ef343e94c831d1551c9b5157128000d0 not found: ID does not exist"
Sep 10 00:18:05.271876 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.271855    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1d36d9-68f47dff7d-r2lx6"]
Sep 10 00:18:05.284379 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.284358    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:18:05.286630 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:05.286611    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1d36d9-76c84c4f5-l6jkg"]
Sep 10 00:18:07.202580 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:07.202530    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" path="/var/lib/kubelet/pods/05ebc238-edf1-4c4e-bf97-cd4551887c6b/volumes"
Sep 10 00:18:07.202977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:07.202919    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" path="/var/lib/kubelet/pods/b1d3c2cc-a437-4135-a8e8-070eeabec58a/volumes"
Sep 10 00:18:38.142630 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:38.142591    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:18:48.819290 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:48.819256    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:18:48.820207 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:48.820153    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" podUID="0a714034-64a0-4635-a581-c0b2aa8fb150" containerName="maas-api" containerID="cri-o://3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f" gracePeriod=30
Sep 10 00:18:48.849499 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:48.849466    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"]
Sep 10 00:18:48.851881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:48.851857    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-01b7d5-29816655-dgktv"]
Sep 10 00:18:49.063615 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.063589    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:18:49.118399 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.118362    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs\") pod \"0a714034-64a0-4635-a581-c0b2aa8fb150\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") "
Sep 10 00:18:49.118594 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.118442    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca\") pod \"0a714034-64a0-4635-a581-c0b2aa8fb150\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") "
Sep 10 00:18:49.118594 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.118467    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls\") pod \"0a714034-64a0-4635-a581-c0b2aa8fb150\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") "
Sep 10 00:18:49.118594 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.118519    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b5p9\" (UniqueName: \"kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9\") pod \"0a714034-64a0-4635-a581-c0b2aa8fb150\" (UID: \"0a714034-64a0-4635-a581-c0b2aa8fb150\") "
Sep 10 00:18:49.118860 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.118831    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a714034-64a0-4635-a581-c0b2aa8fb150" (UID: "0a714034-64a0-4635-a581-c0b2aa8fb150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:49.120515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.120484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a714034-64a0-4635-a581-c0b2aa8fb150" (UID: "0a714034-64a0-4635-a581-c0b2aa8fb150"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:49.120718 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.120695    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a714034-64a0-4635-a581-c0b2aa8fb150" (UID: "0a714034-64a0-4635-a581-c0b2aa8fb150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:49.120794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.120779    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9" (OuterVolumeSpecName: "kube-api-access-7b5p9") pod "0a714034-64a0-4635-a581-c0b2aa8fb150" (UID: "0a714034-64a0-4635-a581-c0b2aa8fb150"). InnerVolumeSpecName "kube-api-access-7b5p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:49.203565 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.203510    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a326325-c11f-42bb-a290-1f48e7a33260" path="/var/lib/kubelet/pods/5a326325-c11f-42bb-a290-1f48e7a33260/volumes"
Sep 10 00:18:49.219210 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.219185    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.219210 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.219210    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a714034-64a0-4635-a581-c0b2aa8fb150-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.219356 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.219218    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a714034-64a0-4635-a581-c0b2aa8fb150-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.219356 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.219227    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5p9\" (UniqueName: \"kubernetes.io/projected/0a714034-64a0-4635-a581-c0b2aa8fb150-kube-api-access-7b5p9\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.504064 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.503975    2576 generic.go:356] "Generic (PLEG): container finished" podID="0a714034-64a0-4635-a581-c0b2aa8fb150" containerID="3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f" exitCode=0
Sep 10 00:18:49.504064 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.504018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"
Sep 10 00:18:49.504274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.504063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" event={"ID":"0a714034-64a0-4635-a581-c0b2aa8fb150","Type":"ContainerDied","Data":"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"}
Sep 10 00:18:49.504274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.504118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp" event={"ID":"0a714034-64a0-4635-a581-c0b2aa8fb150","Type":"ContainerDied","Data":"34cf9b0f75976b8f7da7078248c6e766dab1571ee5b022371540da89d9c6e65d"}
Sep 10 00:18:49.504274 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.504141    2576 scope.go:117] "RemoveContainer" containerID="3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"
Sep 10 00:18:49.515297 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.515270    2576 scope.go:117] "RemoveContainer" containerID="3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"
Sep 10 00:18:49.516206 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:18:49.516160    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f\": container with ID starting with 3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f not found: ID does not exist" containerID="3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"
Sep 10 00:18:49.516316 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.516236    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f"} err="failed to get container status \"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f\": rpc error: code = NotFound desc = could not find container \"3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f\": container with ID starting with 3e8a597218d7ebf37cc40609f6351ca775fffec9bc0fb5b60f06d91dead6a68f not found: ID does not exist"
Sep 10 00:18:49.530653 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.530629    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:18:49.532859 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:49.532839    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-01b7d5-64dcfb567f-lhtbp"]
Sep 10 00:18:51.203382 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:51.203352    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a714034-64a0-4635-a581-c0b2aa8fb150" path="/var/lib/kubelet/pods/0a714034-64a0-4635-a581-c0b2aa8fb150/volumes"
Sep 10 00:18:53.163609 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.163553    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" podUID="1753370f-3aec-4b76-ba84-1add86a8002e" containerName="main" containerID="cri-o://83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b" gracePeriod=45
Sep 10 00:18:53.428825 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.428766    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:18:53.457800 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.457772    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.457968 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.457811    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.457968 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.457871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.457968 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.457938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.458151 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.457993    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c4k7\" (UniqueName: \"kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.458151 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm\") pod \"1753370f-3aec-4b76-ba84-1add86a8002e\" (UID: \"1753370f-3aec-4b76-ba84-1add86a8002e\") "
Sep 10 00:18:53.458151 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458056    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.458151 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458087    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.458377 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458353    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.458444 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458381    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.458532 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.458504    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home" (OuterVolumeSpecName: "home") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.460365 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.460336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm" (OuterVolumeSpecName: "dshm") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.460365 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.460355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7" (OuterVolumeSpecName: "kube-api-access-7c4k7") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "kube-api-access-7c4k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.460491 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.460412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1753370f-3aec-4b76-ba84-1add86a8002e" (UID: "1753370f-3aec-4b76-ba84-1add86a8002e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.529589 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.529529    2576 generic.go:356] "Generic (PLEG): container finished" podID="1753370f-3aec-4b76-ba84-1add86a8002e" containerID="83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b" exitCode=0
Sep 10 00:18:53.529775 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.529599    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"
Sep 10 00:18:53.529775 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.529653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" event={"ID":"1753370f-3aec-4b76-ba84-1add86a8002e","Type":"ContainerDied","Data":"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"}
Sep 10 00:18:53.529775 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.529704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9" event={"ID":"1753370f-3aec-4b76-ba84-1add86a8002e","Type":"ContainerDied","Data":"b76c41d2cee2bc3652f8105dc28f10923b17c8fa1c1d6ea0750ca5a2be88e261"}
Sep 10 00:18:53.529775 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.529721    2576 scope.go:117] "RemoveContainer" containerID="83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"
Sep 10 00:18:53.541299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.541277    2576 scope.go:117] "RemoveContainer" containerID="83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"
Sep 10 00:18:53.541669 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:18:53.541641    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b\": container with ID starting with 83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b not found: ID does not exist" containerID="83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"
Sep 10 00:18:53.541744 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.541686    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b"} err="failed to get container status \"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b\": rpc error: code = NotFound desc = could not find container \"83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b\": container with ID starting with 83ffdd92688eeb25a6dd7dd11bfb485611b66d8179fcbd81751594afb7980d1b not found: ID does not exist"
Sep 10 00:18:53.554931 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.554895    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:18:53.557698 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.557677    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-2-w0-01b7d5-kserve-6b975bb559-g29d9"]
Sep 10 00:18:53.559090 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.559068    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4k7\" (UniqueName: \"kubernetes.io/projected/1753370f-3aec-4b76-ba84-1add86a8002e-kube-api-access-7c4k7\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.559186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.559093    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.559186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.559107    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753370f-3aec-4b76-ba84-1add86a8002e-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.559186 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:53.559117    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753370f-3aec-4b76-ba84-1add86a8002e-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:18:55.203860 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:18:55.203826    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1753370f-3aec-4b76-ba84-1add86a8002e" path="/var/lib/kubelet/pods/1753370f-3aec-4b76-ba84-1add86a8002e/volumes"
Sep 10 00:19:39.090812 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:39.090780    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:19:39.124268 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:39.124229    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:19:49.742843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:49.742811    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:19:49.743325 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:49.743021    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" podUID="1037f845-0203-4257-80dc-5f0e8b084dc8" containerName="maas-api" containerID="cri-o://37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b" gracePeriod=30
Sep 10 00:19:49.993384 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:49.993319    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:19:50.086346 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.086305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs\") pod \"1037f845-0203-4257-80dc-5f0e8b084dc8\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") "
Sep 10 00:19:50.086514 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.086369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8krr\" (UniqueName: \"kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr\") pod \"1037f845-0203-4257-80dc-5f0e8b084dc8\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") "
Sep 10 00:19:50.086514 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.086388    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") pod \"1037f845-0203-4257-80dc-5f0e8b084dc8\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") "
Sep 10 00:19:50.086514 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.086432    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca\") pod \"1037f845-0203-4257-80dc-5f0e8b084dc8\" (UID: \"1037f845-0203-4257-80dc-5f0e8b084dc8\") "
Sep 10 00:19:50.086871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.086844    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1037f845-0203-4257-80dc-5f0e8b084dc8" (UID: "1037f845-0203-4257-80dc-5f0e8b084dc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:19:50.088596 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.088565    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1037f845-0203-4257-80dc-5f0e8b084dc8" (UID: "1037f845-0203-4257-80dc-5f0e8b084dc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:50.088702 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.088598    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1037f845-0203-4257-80dc-5f0e8b084dc8" (UID: "1037f845-0203-4257-80dc-5f0e8b084dc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:50.088702 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.088611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr" (OuterVolumeSpecName: "kube-api-access-x8krr") pod "1037f845-0203-4257-80dc-5f0e8b084dc8" (UID: "1037f845-0203-4257-80dc-5f0e8b084dc8"). InnerVolumeSpecName "kube-api-access-x8krr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:50.187920 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.187888    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.187920 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.187914    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8krr\" (UniqueName: \"kubernetes.io/projected/1037f845-0203-4257-80dc-5f0e8b084dc8-kube-api-access-x8krr\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.187920 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.187924    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1037f845-0203-4257-80dc-5f0e8b084dc8-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.188135 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.187932    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1037f845-0203-4257-80dc-5f0e8b084dc8-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.847162 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.847121    2576 generic.go:356] "Generic (PLEG): container finished" podID="1037f845-0203-4257-80dc-5f0e8b084dc8" containerID="37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b" exitCode=0
Sep 10 00:19:50.847599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.847170    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"
Sep 10 00:19:50.847599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.847177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" event={"ID":"1037f845-0203-4257-80dc-5f0e8b084dc8","Type":"ContainerDied","Data":"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"}
Sep 10 00:19:50.847599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.847244    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb" event={"ID":"1037f845-0203-4257-80dc-5f0e8b084dc8","Type":"ContainerDied","Data":"7f9a326773fa3c4e4e88a7936504ff0d062deb5527ae4559e0bc5f38749829d5"}
Sep 10 00:19:50.847599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.847270    2576 scope.go:117] "RemoveContainer" containerID="37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"
Sep 10 00:19:50.857414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.857392    2576 scope.go:117] "RemoveContainer" containerID="37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"
Sep 10 00:19:50.857733 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:19:50.857712    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b\": container with ID starting with 37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b not found: ID does not exist" containerID="37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"
Sep 10 00:19:50.857809 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.857743    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b"} err="failed to get container status \"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b\": rpc error: code = NotFound desc = could not find container \"37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b\": container with ID starting with 37a3d2d4bb5728243605b69dc547c132e8cdf15e841044cfbe5665b0b4e2347b not found: ID does not exist"
Sep 10 00:19:50.873014 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.872989    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:19:50.875320 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:50.875299    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-cc0040-58cbb9c556-2v2nb"]
Sep 10 00:19:51.203109 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:51.203031    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1037f845-0203-4257-80dc-5f0e8b084dc8" path="/var/lib/kubelet/pods/1037f845-0203-4257-80dc-5f0e8b084dc8/volumes"
Sep 10 00:19:54.111467 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.111420    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" podUID="67bedfda-a6d7-4480-b3f8-6854f6977975" containerName="main" containerID="cri-o://6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91" gracePeriod=45
Sep 10 00:19:54.143977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.143941    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" podUID="d64469fe-af00-4922-a491-53156e92881a" containerName="main" containerID="cri-o://8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2" gracePeriod=45
Sep 10 00:19:54.437645 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.437615    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:19:54.441800 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.441776    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:19:54.528332 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528343    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528409    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528501    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528533 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528521    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528621    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q8ll\" (UniqueName: \"kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll\") pod \"67bedfda-a6d7-4480-b3f8-6854f6977975\" (UID: \"67bedfda-a6d7-4480-b3f8-6854f6977975\") "
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528665    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home" (OuterVolumeSpecName: "home") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528690    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ptvc\" (UniqueName: \"kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc\") pod \"d64469fe-af00-4922-a491-53156e92881a\" (UID: \"d64469fe-af00-4922-a491-53156e92881a\") "
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.528910 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.528835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home" (OuterVolumeSpecName: "home") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.529288 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.529288 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529121    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.529288 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529139    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.529288 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529153    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.529288 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529165    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.529785 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache" (OuterVolumeSpecName: "model-cache") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.529785 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.529490    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.531161 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531134    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll" (OuterVolumeSpecName: "kube-api-access-4q8ll") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "kube-api-access-4q8ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:54.531161 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531143    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc" (OuterVolumeSpecName: "kube-api-access-4ptvc") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "kube-api-access-4ptvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:54.531295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531216    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:54.531295 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531274    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:54.531411 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531367    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm" (OuterVolumeSpecName: "dshm") pod "d64469fe-af00-4922-a491-53156e92881a" (UID: "d64469fe-af00-4922-a491-53156e92881a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.531534 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.531517    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm" (OuterVolumeSpecName: "dshm") pod "67bedfda-a6d7-4480-b3f8-6854f6977975" (UID: "67bedfda-a6d7-4480-b3f8-6854f6977975"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.629886 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629853    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.629886 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629883    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67bedfda-a6d7-4480-b3f8-6854f6977975-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629898    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q8ll\" (UniqueName: \"kubernetes.io/projected/67bedfda-a6d7-4480-b3f8-6854f6977975-kube-api-access-4q8ll\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629911    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629924    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ptvc\" (UniqueName: \"kubernetes.io/projected/d64469fe-af00-4922-a491-53156e92881a-kube-api-access-4ptvc\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629934    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67bedfda-a6d7-4480-b3f8-6854f6977975-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629944    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d64469fe-af00-4922-a491-53156e92881a-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.630102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.629953    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d64469fe-af00-4922-a491-53156e92881a-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.872900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.872862    2576 generic.go:356] "Generic (PLEG): container finished" podID="d64469fe-af00-4922-a491-53156e92881a" containerID="8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2" exitCode=0
Sep 10 00:19:54.873083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.872910    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"
Sep 10 00:19:54.873083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.872951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" event={"ID":"d64469fe-af00-4922-a491-53156e92881a","Type":"ContainerDied","Data":"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"}
Sep 10 00:19:54.873083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.872997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj" event={"ID":"d64469fe-af00-4922-a491-53156e92881a","Type":"ContainerDied","Data":"997398e57698fe5d4a2e8efeddc53abf4ce7247d23131bc11884965d7df02fcd"}
Sep 10 00:19:54.873083 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.873025    2576 scope.go:117] "RemoveContainer" containerID="8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"
Sep 10 00:19:54.874512 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.874491    2576 generic.go:356] "Generic (PLEG): container finished" podID="67bedfda-a6d7-4480-b3f8-6854f6977975" containerID="6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91" exitCode=0
Sep 10 00:19:54.874623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.874565    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"
Sep 10 00:19:54.874623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.874571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" event={"ID":"67bedfda-a6d7-4480-b3f8-6854f6977975","Type":"ContainerDied","Data":"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"}
Sep 10 00:19:54.874623 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.874601    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k" event={"ID":"67bedfda-a6d7-4480-b3f8-6854f6977975","Type":"ContainerDied","Data":"01d06fa085ae108505dbeb949dbe32986620df9ec43f5ffa1e4f72ca881ebd6f"}
Sep 10 00:19:54.883717 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.883696    2576 scope.go:117] "RemoveContainer" containerID="8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"
Sep 10 00:19:54.884019 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:19:54.883998    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2\": container with ID starting with 8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2 not found: ID does not exist" containerID="8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"
Sep 10 00:19:54.884115 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.884030    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2"} err="failed to get container status \"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2\": rpc error: code = NotFound desc = could not find container \"8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2\": container with ID starting with 8f760df3a57a7eb2d79c77a9b493e74919ed339d71674913cc5a5d15198dc9d2 not found: ID does not exist"
Sep 10 00:19:54.884115 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.884053    2576 scope.go:117] "RemoveContainer" containerID="6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"
Sep 10 00:19:54.892603 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.892580    2576 scope.go:117] "RemoveContainer" containerID="6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"
Sep 10 00:19:54.892855 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:19:54.892837    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91\": container with ID starting with 6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91 not found: ID does not exist" containerID="6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"
Sep 10 00:19:54.892923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.892860    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91"} err="failed to get container status \"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91\": rpc error: code = NotFound desc = could not find container \"6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91\": container with ID starting with 6cb5c1104733ec843536f0df29a5b7ac37542db5060c3d3eda8f1f5a6b655c91 not found: ID does not exist"
Sep 10 00:19:54.894868 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.894846    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:19:54.900497 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.900474    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/simulator-w4-cc0040-kserve-7bb5dcc5dc-f6hwj"]
Sep 10 00:19:54.910563 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.910524    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:19:54.916263 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:54.916237    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-w4-cc0040-kserve-54496455fc-ksh4k"]
Sep 10 00:19:55.203225 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:55.203186    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bedfda-a6d7-4480-b3f8-6854f6977975" path="/var/lib/kubelet/pods/67bedfda-a6d7-4480-b3f8-6854f6977975/volumes"
Sep 10 00:19:55.203715 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:19:55.203574    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d64469fe-af00-4922-a491-53156e92881a" path="/var/lib/kubelet/pods/d64469fe-af00-4922-a491-53156e92881a/volumes"
Sep 10 00:20:05.732518 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:05.732481    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:20:07.444769 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:07.444736    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:20:20.752923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:20.752824    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" podUID="8b79226d-703d-4e60-a11d-d475794018f9" containerName="main" containerID="cri-o://4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a" gracePeriod=45
Sep 10 00:20:21.000926 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.000893    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:20:21.024290 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.024219    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b79226d-703d-4e60-a11d-d475794018f9" containerID="4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a" exitCode=0
Sep 10 00:20:21.024290 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.024259    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"
Sep 10 00:20:21.024507 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.024300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" event={"ID":"8b79226d-703d-4e60-a11d-d475794018f9","Type":"ContainerDied","Data":"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"}
Sep 10 00:20:21.024507 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.024338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp" event={"ID":"8b79226d-703d-4e60-a11d-d475794018f9","Type":"ContainerDied","Data":"c5b9b29a904c87a266e1ce6dbd728730bc49a735752ee365567c68d891ee25b3"}
Sep 10 00:20:21.024507 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.024358    2576 scope.go:117] "RemoveContainer" containerID="4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"
Sep 10 00:20:21.035057 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.035038    2576 scope.go:117] "RemoveContainer" containerID="4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"
Sep 10 00:20:21.035349 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:20:21.035331    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a\": container with ID starting with 4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a not found: ID does not exist" containerID="4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"
Sep 10 00:20:21.035404 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.035358    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a"} err="failed to get container status \"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a\": rpc error: code = NotFound desc = could not find container \"4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a\": container with ID starting with 4f1efbf250347ca12dae6b7310815ab6b07b57b7c1c845b0e66bf9f58843fa1a not found: ID does not exist"
Sep 10 00:20:21.089091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089196 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089196 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089196 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089154    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089196 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjd8c\" (UniqueName: \"kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089398 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home\") pod \"8b79226d-703d-4e60-a11d-d475794018f9\" (UID: \"8b79226d-703d-4e60-a11d-d475794018f9\") "
Sep 10 00:20:21.089460 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089373    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:21.089519 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:21.089611 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.089577    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home" (OuterVolumeSpecName: "home") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:21.091487 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.091465    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:20:21.091568 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.091466    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm" (OuterVolumeSpecName: "dshm") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:21.091618 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.091522    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c" (OuterVolumeSpecName: "kube-api-access-qjd8c") pod "8b79226d-703d-4e60-a11d-d475794018f9" (UID: "8b79226d-703d-4e60-a11d-d475794018f9"). InnerVolumeSpecName "kube-api-access-qjd8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:20:21.189859 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189831    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.189859 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189854    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjd8c\" (UniqueName: \"kubernetes.io/projected/8b79226d-703d-4e60-a11d-d475794018f9-kube-api-access-qjd8c\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.189994 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189864    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.189994 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189872    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.189994 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189879    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b79226d-703d-4e60-a11d-d475794018f9-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.189994 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.189888    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b79226d-703d-4e60-a11d-d475794018f9-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:21.339814 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.339783    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:20:21.345088 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:21.345063    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/auth-test-model-1de9a7-kserve-57d9c754dc-6cpqp"]
Sep 10 00:20:22.465937 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.465901    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" podUID="e7938db7-5837-4808-9450-45a07ba8f258" containerName="main" containerID="cri-o://53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e" gracePeriod=45
Sep 10 00:20:22.723860 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.723802    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:20:22.805149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805117    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805162    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpzvb\" (UniqueName: \"kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805603 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805347    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm\") pod \"e7938db7-5837-4808-9450-45a07ba8f258\" (UID: \"e7938db7-5837-4808-9450-45a07ba8f258\") "
Sep 10 00:20:22.805603 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:22.805603 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:22.805752 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805663    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home" (OuterVolumeSpecName: "home") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:22.805752 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805718    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:22.805752 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.805739    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:22.807678 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.807643    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm" (OuterVolumeSpecName: "dshm") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:20:22.807794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.807749    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:20:22.807794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.807754    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb" (OuterVolumeSpecName: "kube-api-access-cpzvb") pod "e7938db7-5837-4808-9450-45a07ba8f258" (UID: "e7938db7-5837-4808-9450-45a07ba8f258"). InnerVolumeSpecName "kube-api-access-cpzvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:20:22.906632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.906592    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpzvb\" (UniqueName: \"kubernetes.io/projected/e7938db7-5837-4808-9450-45a07ba8f258-kube-api-access-cpzvb\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:22.906632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.906627    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:22.906632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.906637    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7938db7-5837-4808-9450-45a07ba8f258-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:22.906632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:22.906644    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7938db7-5837-4808-9450-45a07ba8f258-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:20:23.037871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.037771    2576 generic.go:356] "Generic (PLEG): container finished" podID="e7938db7-5837-4808-9450-45a07ba8f258" containerID="53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e" exitCode=0
Sep 10 00:20:23.037871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.037820    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"
Sep 10 00:20:23.037871 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.037856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" event={"ID":"e7938db7-5837-4808-9450-45a07ba8f258","Type":"ContainerDied","Data":"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"}
Sep 10 00:20:23.038160 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.037898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6" event={"ID":"e7938db7-5837-4808-9450-45a07ba8f258","Type":"ContainerDied","Data":"5004332664cb81296bd91a12a4db262c3b989f8c6737329056cfc1d9627a3f8f"}
Sep 10 00:20:23.038160 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.037916    2576 scope.go:117] "RemoveContainer" containerID="53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"
Sep 10 00:20:23.047695 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.047679    2576 scope.go:117] "RemoveContainer" containerID="53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"
Sep 10 00:20:23.047993 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:20:23.047972    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e\": container with ID starting with 53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e not found: ID does not exist" containerID="53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"
Sep 10 00:20:23.048101 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.048000    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e"} err="failed to get container status \"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e\": rpc error: code = NotFound desc = could not find container \"53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e\": container with ID starting with 53f8a41d8cfe76b868c2690311b0e12d16f2052bb08e41b5a87fb86cf1a2167e not found: ID does not exist"
Sep 10 00:20:23.059084 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.059058    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:20:23.061212 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.061193    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/auth-test-model-b1baac-kserve-f8486db95-j4qk6"]
Sep 10 00:20:23.202300 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.202268    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b79226d-703d-4e60-a11d-d475794018f9" path="/var/lib/kubelet/pods/8b79226d-703d-4e60-a11d-d475794018f9/volumes"
Sep 10 00:20:23.202712 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:20:23.202694    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7938db7-5837-4808-9450-45a07ba8f258" path="/var/lib/kubelet/pods/e7938db7-5837-4808-9450-45a07ba8f258/volumes"
Sep 10 00:21:11.995880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.995842    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7792-00b521"]
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996179    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5446661-0def-44a9-8219-b352090db0c3" containerName="payload-pre-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996190    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5446661-0def-44a9-8219-b352090db0c3" containerName="payload-pre-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996200    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df8474d-3c76-4c31-ad84-3489fd173307" containerName="payload-pre-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996206    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df8474d-3c76-4c31-ad84-3489fd173307" containerName="payload-pre-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996214    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1947068-fe5a-46a3-827e-a0c4318a86c3" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996219    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1947068-fe5a-46a3-827e-a0c4318a86c3" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996227    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996232    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996241    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1753370f-3aec-4b76-ba84-1add86a8002e" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996246    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1753370f-3aec-4b76-ba84-1add86a8002e" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996251    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80617e9-27a2-4395-b560-2255a521d04d" containerName="payload-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996256    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80617e9-27a2-4395-b560-2255a521d04d" containerName="payload-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996262    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" containerName="payload-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996268    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" containerName="payload-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996273    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7938db7-5837-4808-9450-45a07ba8f258" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996278    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7938db7-5837-4808-9450-45a07ba8f258" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996288    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1037f845-0203-4257-80dc-5f0e8b084dc8" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996293    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1037f845-0203-4257-80dc-5f0e8b084dc8" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a714034-64a0-4635-a581-c0b2aa8fb150" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996303    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a714034-64a0-4635-a581-c0b2aa8fb150" containerName="maas-api"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996308    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b79226d-703d-4e60-a11d-d475794018f9" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996313    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b79226d-703d-4e60-a11d-d475794018f9" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996318    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bedfda-a6d7-4480-b3f8-6854f6977975" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996323    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bedfda-a6d7-4480-b3f8-6854f6977975" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996329    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64469fe-af00-4922-a491-53156e92881a" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996334    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64469fe-af00-4922-a491-53156e92881a" containerName="main"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996339    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" containerName="payload-pre-processing"
Sep 10 00:21:11.996342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996344    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" containerName="payload-pre-processing"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996397    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80617e9-27a2-4395-b560-2255a521d04d" containerName="payload-processing"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996404    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ebc238-edf1-4c4e-bf97-cd4551887c6b" containerName="payload-processing"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996411    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64469fe-af00-4922-a491-53156e92881a" containerName="main"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996417    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b79226d-703d-4e60-a11d-d475794018f9" containerName="main"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996422    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe7a4ae-1feb-41e1-9235-dd9a9825359d" containerName="maas-api"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996427    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d3c2cc-a437-4135-a8e8-070eeabec58a" containerName="payload-pre-processing"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996433    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7938db7-5837-4808-9450-45a07ba8f258" containerName="main"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996439    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1037f845-0203-4257-80dc-5f0e8b084dc8" containerName="maas-api"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996444    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a714034-64a0-4635-a581-c0b2aa8fb150" containerName="maas-api"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996450    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1947068-fe5a-46a3-827e-a0c4318a86c3" containerName="maas-api"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996455    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bedfda-a6d7-4480-b3f8-6854f6977975" containerName="main"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996460    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5446661-0def-44a9-8219-b352090db0c3" containerName="payload-pre-processing"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996464    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1753370f-3aec-4b76-ba84-1add86a8002e" containerName="main"
Sep 10 00:21:11.997334 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.996468    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df8474d-3c76-4c31-ad84-3489fd173307" containerName="payload-pre-processing"
Sep 10 00:21:11.999450 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:11.999436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:12.001935 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.001914    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gtwx5\""
Sep 10 00:21:12.003518 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.003495    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-00b521"]
Sep 10 00:21:12.126014 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.125983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25b82\" (UniqueName: \"kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82\") pod \"test-curl-7792-00b521\" (UID: \"fb9ec8be-4396-42c0-a9fd-185cb7de52d6\") " pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:12.226444 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.226410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25b82\" (UniqueName: \"kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82\") pod \"test-curl-7792-00b521\" (UID: \"fb9ec8be-4396-42c0-a9fd-185cb7de52d6\") " pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:12.234331 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.234303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25b82\" (UniqueName: \"kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82\") pod \"test-curl-7792-00b521\" (UID: \"fb9ec8be-4396-42c0-a9fd-185cb7de52d6\") " pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:12.310516 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.310446    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:12.578423 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:12.578399    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-00b521"]
Sep 10 00:21:12.579902 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:21:12.579874    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb9ec8be_4396_42c0_a9fd_185cb7de52d6.slice/crio-73aa4122722193801ed610f62fb362d66da1c71373ea53f04fd192c1a8e84f00 WatchSource:0}: Error finding container 73aa4122722193801ed610f62fb362d66da1c71373ea53f04fd192c1a8e84f00: Status 404 returned error can't find the container with id 73aa4122722193801ed610f62fb362d66da1c71373ea53f04fd192c1a8e84f00
Sep 10 00:21:13.309954 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:13.309917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-00b521" event={"ID":"fb9ec8be-4396-42c0-a9fd-185cb7de52d6","Type":"ContainerStarted","Data":"73aa4122722193801ed610f62fb362d66da1c71373ea53f04fd192c1a8e84f00"}
Sep 10 00:21:20.352618 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:20.352581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-00b521" event={"ID":"fb9ec8be-4396-42c0-a9fd-185cb7de52d6","Type":"ContainerStarted","Data":"488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0"}
Sep 10 00:21:20.367885 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:20.367833    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7792-00b521" podStartSLOduration=2.304398817 podStartE2EDuration="9.367817086s" totalImagesPullingTime="7.063418269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:21:20.366182091 +0000 UTC m=+1953.686558168" watchObservedRunningTime="2026-09-10 00:21:20.367817086 +0000 UTC m=+1953.688193150"
Sep 10 00:21:20.953714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:20.953680    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7792-00b521"]
Sep 10 00:21:20.960142 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:20.960108    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7792-00b521"]
Sep 10 00:21:21.357516 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:21:21.357482    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7792-00b521" container="test-curl-7792-00b521"
Sep 10 00:21:22.362170 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:22.362114    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7792-00b521" podUID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" containerName="test-curl-7792-00b521" containerID="cri-o://488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0" gracePeriod=2
Sep 10 00:21:24.539688 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:24.539663    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:24.644164 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:24.644055    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25b82\" (UniqueName: \"kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82\") pod \"fb9ec8be-4396-42c0-a9fd-185cb7de52d6\" (UID: \"fb9ec8be-4396-42c0-a9fd-185cb7de52d6\") "
Sep 10 00:21:24.646246 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:24.646215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82" (OuterVolumeSpecName: "kube-api-access-25b82") pod "fb9ec8be-4396-42c0-a9fd-185cb7de52d6" (UID: "fb9ec8be-4396-42c0-a9fd-185cb7de52d6"). InnerVolumeSpecName "kube-api-access-25b82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:21:24.745240 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:24.745215    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25b82\" (UniqueName: \"kubernetes.io/projected/fb9ec8be-4396-42c0-a9fd-185cb7de52d6-kube-api-access-25b82\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:21:25.202887 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.202856    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" path="/var/lib/kubelet/pods/fb9ec8be-4396-42c0-a9fd-185cb7de52d6/volumes"
Sep 10 00:21:25.379505 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.379469    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" containerID="488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0" exitCode=137
Sep 10 00:21:25.379505 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.379510    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-00b521"
Sep 10 00:21:25.379725 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.379576    2576 scope.go:117] "RemoveContainer" containerID="488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0"
Sep 10 00:21:25.388459 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.388445    2576 scope.go:117] "RemoveContainer" containerID="488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0"
Sep 10 00:21:25.388736 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:21:25.388716    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0\": container with ID starting with 488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0 not found: ID does not exist" containerID="488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0"
Sep 10 00:21:25.388810 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:25.388750    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0"} err="failed to get container status \"488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0\": rpc error: code = NotFound desc = could not find container \"488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0\": container with ID starting with 488a5af3c1c806aed7f9b400d7e760076fad9ac5e14809b1a5c14cece1be66f0 not found: ID does not exist"
Sep 10 00:21:37.954332 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.954299    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:21:37.954902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.954721    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" containerName="test-curl-7792-00b521"
Sep 10 00:21:37.954902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.954739    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" containerName="test-curl-7792-00b521"
Sep 10 00:21:37.954902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.954838    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9ec8be-4396-42c0-a9fd-185cb7de52d6" containerName="test-curl-7792-00b521"
Sep 10 00:21:37.959098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.959079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:37.961613 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.961590    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"openshift-service-ca.crt\""
Sep 10 00:21:37.962747 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.962724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"default-dockercfg-bkdjc\""
Sep 10 00:21:37.962873 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.962801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"rate-test-model-1de9a7-kserve-self-signed-certs\""
Sep 10 00:21:37.962873 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.962846    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"kube-root-ca.crt\""
Sep 10 00:21:37.976409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:37.976381    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:21:38.055059 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.055059 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpn4\" (UniqueName: \"kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.055275 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.055275 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055096    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.055275 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.055379 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.055280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.155830 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.155778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpn4\" (UniqueName: \"kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.155830 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.155832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.155858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.155980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.156003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156091 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.156045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156299 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.156236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156377 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.156348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.156440 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.156396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.158460 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.158439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.158611 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.158596    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.168620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.168603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpn4\" (UniqueName: \"kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4\") pod \"rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.278123 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.278040    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:38.549799 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:38.549770    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:21:38.550941 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:21:38.550913    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1b3125f_98b2_4ef8_adb3_6322a971e165.slice/crio-d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9 WatchSource:0}: Error finding container d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9: Status 404 returned error can't find the container with id d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9
Sep 10 00:21:39.455947 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:39.455913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" event={"ID":"d1b3125f-98b2-4ef8-adb3-6322a971e165","Type":"ContainerStarted","Data":"547f52b18b4dc98894eb7ca66a231078961c74f222734b69749514aaa2567fa4"}
Sep 10 00:21:39.455947 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:39.455952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" event={"ID":"d1b3125f-98b2-4ef8-adb3-6322a971e165","Type":"ContainerStarted","Data":"d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9"}
Sep 10 00:21:39.472597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:39.472532    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" podStartSLOduration=2.472516403 podStartE2EDuration="2.472516403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:21:39.471202594 +0000 UTC m=+1972.791578671" watchObservedRunningTime="2026-09-10 00:21:39.472516403 +0000 UTC m=+1972.792892466"
Sep 10 00:21:47.558913 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:47.558878    2576 scope.go:117] "RemoveContainer" containerID="1896156476a78b2027114113c1cbbaff81f92f5f0cfb3cce4cc1668fe2d9e86c"
Sep 10 00:21:48.278948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:48.278908    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:48.278948 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:48.278954    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:48.291147 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:48.291123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:21:48.515069 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:21:48.515035    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:22:44.626828 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:44.626794    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:22:59.647099 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.647028    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" podUID="d1b3125f-98b2-4ef8-adb3-6322a971e165" containerName="main" containerID="cri-o://547f52b18b4dc98894eb7ca66a231078961c74f222734b69749514aaa2567fa4" gracePeriod=45
Sep 10 00:22:59.875558 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.875506    2576 generic.go:356] "Generic (PLEG): container finished" podID="d1b3125f-98b2-4ef8-adb3-6322a971e165" containerID="547f52b18b4dc98894eb7ca66a231078961c74f222734b69749514aaa2567fa4" exitCode=0
Sep 10 00:22:59.875669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.875570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" event={"ID":"d1b3125f-98b2-4ef8-adb3-6322a971e165","Type":"ContainerDied","Data":"547f52b18b4dc98894eb7ca66a231078961c74f222734b69749514aaa2567fa4"}
Sep 10 00:22:59.875669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.875610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz" event={"ID":"d1b3125f-98b2-4ef8-adb3-6322a971e165","Type":"ContainerDied","Data":"d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9"}
Sep 10 00:22:59.875669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.875622    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ad988e5e6f7f6e8c92f5b51ba9159ab358e762770f99cf03e5fb1e8612e1b9"
Sep 10 00:22:59.886404 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.886387    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:22:59.901885 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.901816    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.901989 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.901932    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.901989 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.901960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.901989 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.901984    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.902149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902033    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.902149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902086    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tpn4\" (UniqueName: \"kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4\") pod \"d1b3125f-98b2-4ef8-adb3-6322a971e165\" (UID: \"d1b3125f-98b2-4ef8-adb3-6322a971e165\") "
Sep 10 00:22:59.902251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902149    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:22:59.902251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902220    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:22:59.902364 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home" (OuterVolumeSpecName: "home") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:22:59.902414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902366    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:22:59.902414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902382    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:22:59.902414 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.902395    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:22:59.905106 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.905073    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm" (OuterVolumeSpecName: "dshm") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:22:59.905260 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.905219    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:22:59.905389 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:22:59.905300    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4" (OuterVolumeSpecName: "kube-api-access-7tpn4") pod "d1b3125f-98b2-4ef8-adb3-6322a971e165" (UID: "d1b3125f-98b2-4ef8-adb3-6322a971e165"). InnerVolumeSpecName "kube-api-access-7tpn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:23:00.003257 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.003233    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b3125f-98b2-4ef8-adb3-6322a971e165-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.003257 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.003256    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b3125f-98b2-4ef8-adb3-6322a971e165-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.003430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.003264    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpn4\" (UniqueName: \"kubernetes.io/projected/d1b3125f-98b2-4ef8-adb3-6322a971e165-kube-api-access-7tpn4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.881621 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.881585    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"
Sep 10 00:23:00.901957 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.901926    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:23:00.904029 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:00.904008    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/rate-test-model-1de9a7-kserve-74cfc79cf9-6hrsz"]
Sep 10 00:23:01.201815 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:01.201739    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b3125f-98b2-4ef8-adb3-6322a971e165" path="/var/lib/kubelet/pods/d1b3125f-98b2-4ef8-adb3-6322a971e165/volumes"
Sep 10 00:23:19.601262 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.601185    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:23:19.601687 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.601561    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b3125f-98b2-4ef8-adb3-6322a971e165" containerName="main"
Sep 10 00:23:19.601687 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.601573    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b3125f-98b2-4ef8-adb3-6322a971e165" containerName="main"
Sep 10 00:23:19.601687 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.601634    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b3125f-98b2-4ef8-adb3-6322a971e165" containerName="main"
Sep 10 00:23:19.605854 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.605828    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.608314 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.608293    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h47kt\""
Sep 10 00:23:19.608447 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.608293    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c9018-serving-cert\""
Sep 10 00:23:19.616251 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.615076    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:23:19.672429 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.672403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.672585 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.672444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.672585 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.672485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.672674 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.672579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.773191 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.773157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.773344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.773206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.773344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.773243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.773344 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.773271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.773923 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.773900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.775963 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.775939    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.776063 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.775944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.780936 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.780916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh\") pod \"maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:19.920097 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:19.920022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:20.201857 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:20.201829    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:23:20.203690 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:23:20.203666    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9b1424_37ee_4234_8c4c_bc7ddd4e4688.slice/crio-898e40afcea01b6e76da0978c8c15a089fbd235ab606be6d3736e01439d087e7 WatchSource:0}: Error finding container 898e40afcea01b6e76da0978c8c15a089fbd235ab606be6d3736e01439d087e7: Status 404 returned error can't find the container with id 898e40afcea01b6e76da0978c8c15a089fbd235ab606be6d3736e01439d087e7
Sep 10 00:23:20.205099 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:20.205081    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:23:20.996408 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:20.996331    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" event={"ID":"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688","Type":"ContainerStarted","Data":"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"}
Sep 10 00:23:20.996408 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:20.996382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" event={"ID":"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688","Type":"ContainerStarted","Data":"898e40afcea01b6e76da0978c8c15a089fbd235ab606be6d3736e01439d087e7"}
Sep 10 00:23:20.996828 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:20.996431    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:21.012166 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:21.012117    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" podStartSLOduration=1.642932224 podStartE2EDuration="2.012103067s" totalImagesPullingTime="369.170843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:23:21.010820513 +0000 UTC m=+2074.331196574" watchObservedRunningTime="2026-09-10 00:23:21.012103067 +0000 UTC m=+2074.332479133"
Sep 10 00:23:27.007759 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:27.007733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:23:48.707917 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:23:48.707882    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:24:24.714945 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:24.714910    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:24:29.960239 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:29.960202    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:24:29.964283 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:29.964261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:29.966708 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:29.966684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-70937b-serving-cert\""
Sep 10 00:24:29.975826 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:29.975803    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:24:30.077745 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.077714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.077896 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.077755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrr9d\" (UniqueName: \"kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.077896 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.077782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.077896 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.077824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.178935 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.178904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrr9d\" (UniqueName: \"kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.178943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.179074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.178973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.179148 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:24:30.179074    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-serving-cert: secret "maas-api-e2e-ipp-cleanup-70937b-serving-cert" not found
Sep 10 00:24:30.179148 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:24:30.179119    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls podName:20629073-fdfc-421c-ac5d-51d7d26018bf nodeName:}" failed. No retries permitted until 2026-09-10 00:24:30.679105255 +0000 UTC m=+2143.999481296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" (UID: "20629073-fdfc-421c-ac5d-51d7d26018bf") : secret "maas-api-e2e-ipp-cleanup-70937b-serving-cert" not found
Sep 10 00:24:30.179265 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.179159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.179647 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.179618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.181755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.181731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.187386 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.187366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrr9d\" (UniqueName: \"kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.653577 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.653530    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:24:30.657191 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.657176    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.661439 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.661416    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-70937b\""
Sep 10 00:24:30.661562 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.661425    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-70937b-dockercfg-wmrr4\""
Sep 10 00:24:30.669597 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.669575    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:24:30.682906 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.682877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.685449 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.685423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.784380 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.784351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpl2v\" (UniqueName: \"kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.784532 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.784420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.877676 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.877650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:30.885753 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.885721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.885850 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.885798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpl2v\" (UniqueName: \"kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.886593 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.886569    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.893801 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.893776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpl2v\" (UniqueName: \"kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v\") pod \"payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:30.967868 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:30.967836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:31.150552 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.150517    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:24:31.152010 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:24:31.151976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20629073_fdfc_421c_ac5d_51d7d26018bf.slice/crio-063ab34e3732e4e424adf0672120afed7fc786c0d4e85b5b1fae9462caf1316d WatchSource:0}: Error finding container 063ab34e3732e4e424adf0672120afed7fc786c0d4e85b5b1fae9462caf1316d: Status 404 returned error can't find the container with id 063ab34e3732e4e424adf0672120afed7fc786c0d4e85b5b1fae9462caf1316d
Sep 10 00:24:31.239305 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.239281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:24:31.240155 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:24:31.240132    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1994db70_a1ec_4305_b3ea_ecde6df43d45.slice/crio-2cc58de3efc3ab2cdb09c184747cde0e86d830d7877aca79d821159d550b8145 WatchSource:0}: Error finding container 2cc58de3efc3ab2cdb09c184747cde0e86d830d7877aca79d821159d550b8145: Status 404 returned error can't find the container with id 2cc58de3efc3ab2cdb09c184747cde0e86d830d7877aca79d821159d550b8145
Sep 10 00:24:31.362975 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.362935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" event={"ID":"1994db70-a1ec-4305-b3ea-ecde6df43d45","Type":"ContainerStarted","Data":"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"}
Sep 10 00:24:31.363148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.362991    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" event={"ID":"1994db70-a1ec-4305-b3ea-ecde6df43d45","Type":"ContainerStarted","Data":"2cc58de3efc3ab2cdb09c184747cde0e86d830d7877aca79d821159d550b8145"}
Sep 10 00:24:31.363148 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.363020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:24:31.364260 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.364237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" event={"ID":"20629073-fdfc-421c-ac5d-51d7d26018bf","Type":"ContainerStarted","Data":"063ab34e3732e4e424adf0672120afed7fc786c0d4e85b5b1fae9462caf1316d"}
Sep 10 00:24:31.379692 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:31.379647    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" podStartSLOduration=1.379634208 podStartE2EDuration="1.379634208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:24:31.377486547 +0000 UTC m=+2144.697862612" watchObservedRunningTime="2026-09-10 00:24:31.379634208 +0000 UTC m=+2144.700010281"
Sep 10 00:24:32.370818 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:32.370781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" event={"ID":"20629073-fdfc-421c-ac5d-51d7d26018bf","Type":"ContainerStarted","Data":"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"}
Sep 10 00:24:32.371270 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:32.370935    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:32.389241 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:32.389196    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" podStartSLOduration=2.880666923 podStartE2EDuration="3.389181714s" totalImagesPullingTime="508.514791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:24:32.387218306 +0000 UTC m=+2145.707594379" watchObservedRunningTime="2026-09-10 00:24:32.389181714 +0000 UTC m=+2145.709557812"
Sep 10 00:24:38.381002 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:38.380917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:24:42.373172 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:24:42.373134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:25:00.481991 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.481955    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:25:00.484724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.482243    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" podUID="20629073-fdfc-421c-ac5d-51d7d26018bf" containerName="maas-api" containerID="cri-o://276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c" gracePeriod=30
Sep 10 00:25:00.536855 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.536822    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:25:00.537110 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.537070    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" podUID="1994db70-a1ec-4305-b3ea-ecde6df43d45" containerName="payload-pre-processing" containerID="cri-o://36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd" gracePeriod=30
Sep 10 00:25:00.787122 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.787099    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:25:00.818806 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.818784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:25:00.842845 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.842821    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs\") pod \"20629073-fdfc-421c-ac5d-51d7d26018bf\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") "
Sep 10 00:25:00.842984 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.842883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") pod \"20629073-fdfc-421c-ac5d-51d7d26018bf\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") "
Sep 10 00:25:00.842984 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.842928    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrr9d\" (UniqueName: \"kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d\") pod \"20629073-fdfc-421c-ac5d-51d7d26018bf\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") "
Sep 10 00:25:00.843117 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.842996    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca\") pod \"20629073-fdfc-421c-ac5d-51d7d26018bf\" (UID: \"20629073-fdfc-421c-ac5d-51d7d26018bf\") "
Sep 10 00:25:00.843611 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.843533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20629073-fdfc-421c-ac5d-51d7d26018bf" (UID: "20629073-fdfc-421c-ac5d-51d7d26018bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:00.845149 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.845121    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20629073-fdfc-421c-ac5d-51d7d26018bf" (UID: "20629073-fdfc-421c-ac5d-51d7d26018bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:25:00.845262 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.845157    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "20629073-fdfc-421c-ac5d-51d7d26018bf" (UID: "20629073-fdfc-421c-ac5d-51d7d26018bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:25:00.845262 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.845208    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d" (OuterVolumeSpecName: "kube-api-access-vrr9d") pod "20629073-fdfc-421c-ac5d-51d7d26018bf" (UID: "20629073-fdfc-421c-ac5d-51d7d26018bf"). InnerVolumeSpecName "kube-api-access-vrr9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:00.943510 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943485    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume\") pod \"1994db70-a1ec-4305-b3ea-ecde6df43d45\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") "
Sep 10 00:25:00.943634 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943599    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpl2v\" (UniqueName: \"kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v\") pod \"1994db70-a1ec-4305-b3ea-ecde6df43d45\" (UID: \"1994db70-a1ec-4305-b3ea-ecde6df43d45\") "
Sep 10 00:25:00.943885 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943864    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1994db70-a1ec-4305-b3ea-ecde6df43d45" (UID: "1994db70-a1ec-4305-b3ea-ecde6df43d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:00.943927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943872    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:00.943927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943908    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20629073-fdfc-421c-ac5d-51d7d26018bf-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:00.943927 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943918    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrr9d\" (UniqueName: \"kubernetes.io/projected/20629073-fdfc-421c-ac5d-51d7d26018bf-kube-api-access-vrr9d\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:00.944025 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.943928    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20629073-fdfc-421c-ac5d-51d7d26018bf-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:00.945521 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:00.945503    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v" (OuterVolumeSpecName: "kube-api-access-jpl2v") pod "1994db70-a1ec-4305-b3ea-ecde6df43d45" (UID: "1994db70-a1ec-4305-b3ea-ecde6df43d45"). InnerVolumeSpecName "kube-api-access-jpl2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:01.045258 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.045194    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpl2v\" (UniqueName: \"kubernetes.io/projected/1994db70-a1ec-4305-b3ea-ecde6df43d45-kube-api-access-jpl2v\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:01.045258 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.045219    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1994db70-a1ec-4305-b3ea-ecde6df43d45-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:25:01.526136 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.526101    2576 generic.go:356] "Generic (PLEG): container finished" podID="20629073-fdfc-421c-ac5d-51d7d26018bf" containerID="276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c" exitCode=0
Sep 10 00:25:01.526580 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.526151    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"
Sep 10 00:25:01.526580 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.526185    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" event={"ID":"20629073-fdfc-421c-ac5d-51d7d26018bf","Type":"ContainerDied","Data":"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"}
Sep 10 00:25:01.526580 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.526238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2" event={"ID":"20629073-fdfc-421c-ac5d-51d7d26018bf","Type":"ContainerDied","Data":"063ab34e3732e4e424adf0672120afed7fc786c0d4e85b5b1fae9462caf1316d"}
Sep 10 00:25:01.526580 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.526262    2576 scope.go:117] "RemoveContainer" containerID="276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"
Sep 10 00:25:01.527770 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.527748    2576 generic.go:356] "Generic (PLEG): container finished" podID="1994db70-a1ec-4305-b3ea-ecde6df43d45" containerID="36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd" exitCode=0
Sep 10 00:25:01.527848 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.527818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" event={"ID":"1994db70-a1ec-4305-b3ea-ecde6df43d45","Type":"ContainerDied","Data":"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"}
Sep 10 00:25:01.527908 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.527861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs" event={"ID":"1994db70-a1ec-4305-b3ea-ecde6df43d45","Type":"ContainerDied","Data":"2cc58de3efc3ab2cdb09c184747cde0e86d830d7877aca79d821159d550b8145"}
Sep 10 00:25:01.527908 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.527824    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"
Sep 10 00:25:01.535650 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.535618    2576 scope.go:117] "RemoveContainer" containerID="276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"
Sep 10 00:25:01.535891 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:25:01.535875    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c\": container with ID starting with 276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c not found: ID does not exist" containerID="276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"
Sep 10 00:25:01.535949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.535900    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c"} err="failed to get container status \"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c\": rpc error: code = NotFound desc = could not find container \"276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c\": container with ID starting with 276b50240cbdaec77ec952ba492b0ae6ea5864df908ea6a69441eb7de5de003c not found: ID does not exist"
Sep 10 00:25:01.535949 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.535919    2576 scope.go:117] "RemoveContainer" containerID="36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"
Sep 10 00:25:01.544599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.544576    2576 scope.go:117] "RemoveContainer" containerID="36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"
Sep 10 00:25:01.544906 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:25:01.544887    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd\": container with ID starting with 36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd not found: ID does not exist" containerID="36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"
Sep 10 00:25:01.544969 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.544915    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd"} err="failed to get container status \"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd\": rpc error: code = NotFound desc = could not find container \"36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd\": container with ID starting with 36d9928347f35c162158565f11b9b08a5cf19951a10048f4a5f2b3e8458e1ddd not found: ID does not exist"
Sep 10 00:25:01.545137 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.545124    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:25:01.547465 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.547446    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-70937b-8b888ff94-tvhqs"]
Sep 10 00:25:01.555350 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.555320    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:25:01.558729 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:01.558711    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-70937b-6b59774bd4-q8qc2"]
Sep 10 00:25:03.201486 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:03.201450    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1994db70-a1ec-4305-b3ea-ecde6df43d45" path="/var/lib/kubelet/pods/1994db70-a1ec-4305-b3ea-ecde6df43d45/volumes"
Sep 10 00:25:03.201995 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:25:03.201979    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20629073-fdfc-421c-ac5d-51d7d26018bf" path="/var/lib/kubelet/pods/20629073-fdfc-421c-ac5d-51d7d26018bf/volumes"
Sep 10 00:26:30.579483 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.579390    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:26:30.580097 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.579718    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" podUID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" containerName="maas-api" containerID="cri-o://941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32" gracePeriod=30
Sep 10 00:26:30.829180 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.829157    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:26:30.893672 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.893604    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh\") pod \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") "
Sep 10 00:26:30.893672 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.893658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca\") pod \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") "
Sep 10 00:26:30.893881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.893694    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs\") pod \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") "
Sep 10 00:26:30.893881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.893762    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls\") pod \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\" (UID: \"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688\") "
Sep 10 00:26:30.894074 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.894050    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" (UID: "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:30.895854 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.895825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh" (OuterVolumeSpecName: "kube-api-access-2ltmh") pod "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" (UID: "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688"). InnerVolumeSpecName "kube-api-access-2ltmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:26:30.895973 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.895950    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" (UID: "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:26:30.896029 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.895948    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" (UID: "3c9b1424-37ee-4234-8c4c-bc7ddd4e4688"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:26:30.995050 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.995015    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:30.995050 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.995046    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-kube-api-access-2ltmh\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:30.995229 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.995060    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:30.995229 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:30.995072    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:31.009909 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.009877    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" containerID="941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32" exitCode=0
Sep 10 00:26:31.010054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.009928    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"
Sep 10 00:26:31.010054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.009960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" event={"ID":"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688","Type":"ContainerDied","Data":"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"}
Sep 10 00:26:31.010054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.010017    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp" event={"ID":"3c9b1424-37ee-4234-8c4c-bc7ddd4e4688","Type":"ContainerDied","Data":"898e40afcea01b6e76da0978c8c15a089fbd235ab606be6d3736e01439d087e7"}
Sep 10 00:26:31.010054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.010045    2576 scope.go:117] "RemoveContainer" containerID="941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"
Sep 10 00:26:31.020255 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.020236    2576 scope.go:117] "RemoveContainer" containerID="941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"
Sep 10 00:26:31.020553 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:26:31.020520    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32\": container with ID starting with 941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32 not found: ID does not exist" containerID="941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"
Sep 10 00:26:31.020632 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.020568    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32"} err="failed to get container status \"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32\": rpc error: code = NotFound desc = could not find container \"941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32\": container with ID starting with 941c3a2d08c727b8d0387da43a2c1222f2c0bb137cd0d9b13d365dc348fb6d32 not found: ID does not exist"
Sep 10 00:26:31.031245 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.031224    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:26:31.033467 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.033447    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c9018-6d45b7648d-g6mvp"]
Sep 10 00:26:31.202008 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:31.201926    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" path="/var/lib/kubelet/pods/3c9b1424-37ee-4234-8c4c-bc7ddd4e4688/volumes"
Sep 10 00:26:49.061235 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.061202    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:26:49.061761 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.061498    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" podUID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" containerName="payload-processing" containerID="cri-o://99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a" gracePeriod=30
Sep 10 00:26:49.070781 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.070751    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:26:49.071175 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.071129    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" podUID="27583020-b85b-4e47-8cd0-f1022277a989" containerName="payload-pre-processing" containerID="cri-o://48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f" gracePeriod=30
Sep 10 00:26:49.346041 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.346016    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:26:49.348902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.348884    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:26:49.449899 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.449859    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume\") pod \"27583020-b85b-4e47-8cd0-f1022277a989\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") "
Sep 10 00:26:49.450089 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.449947    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume\") pod \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") "
Sep 10 00:26:49.450089 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gdxd\" (UniqueName: \"kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd\") pod \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\" (UID: \"9c5e98b3-2202-4c64-a703-b4187bbe92ad\") "
Sep 10 00:26:49.450217 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450106    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kfmv\" (UniqueName: \"kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv\") pod \"27583020-b85b-4e47-8cd0-f1022277a989\" (UID: \"27583020-b85b-4e47-8cd0-f1022277a989\") "
Sep 10 00:26:49.450308 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450286    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c5e98b3-2202-4c64-a703-b4187bbe92ad" (UID: "9c5e98b3-2202-4c64-a703-b4187bbe92ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:49.450373 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27583020-b85b-4e47-8cd0-f1022277a989" (UID: "27583020-b85b-4e47-8cd0-f1022277a989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:49.450428 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450404    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5e98b3-2202-4c64-a703-b4187bbe92ad-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:49.450428 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.450419    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27583020-b85b-4e47-8cd0-f1022277a989-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:49.452371 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.452336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv" (OuterVolumeSpecName: "kube-api-access-9kfmv") pod "27583020-b85b-4e47-8cd0-f1022277a989" (UID: "27583020-b85b-4e47-8cd0-f1022277a989"). InnerVolumeSpecName "kube-api-access-9kfmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:26:49.452371 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.452361    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd" (OuterVolumeSpecName: "kube-api-access-8gdxd") pod "9c5e98b3-2202-4c64-a703-b4187bbe92ad" (UID: "9c5e98b3-2202-4c64-a703-b4187bbe92ad"). InnerVolumeSpecName "kube-api-access-8gdxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:26:49.551617 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.551579    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kfmv\" (UniqueName: \"kubernetes.io/projected/27583020-b85b-4e47-8cd0-f1022277a989-kube-api-access-9kfmv\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:49.551617 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:49.551609    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gdxd\" (UniqueName: \"kubernetes.io/projected/9c5e98b3-2202-4c64-a703-b4187bbe92ad-kube-api-access-8gdxd\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:26:50.112929 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.112895    2576 generic.go:356] "Generic (PLEG): container finished" podID="27583020-b85b-4e47-8cd0-f1022277a989" containerID="48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f" exitCode=0
Sep 10 00:26:50.113430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.112950    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"
Sep 10 00:26:50.113430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.112976    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" event={"ID":"27583020-b85b-4e47-8cd0-f1022277a989","Type":"ContainerDied","Data":"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"}
Sep 10 00:26:50.113430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.113033    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk" event={"ID":"27583020-b85b-4e47-8cd0-f1022277a989","Type":"ContainerDied","Data":"ee916dd7f7a7eda0cbbbb440fd823d65486020c69544f715708447dd374455aa"}
Sep 10 00:26:50.113430 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.113055    2576 scope.go:117] "RemoveContainer" containerID="48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"
Sep 10 00:26:50.114457 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.114428    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" containerID="99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a" exitCode=0
Sep 10 00:26:50.114592 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.114465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" event={"ID":"9c5e98b3-2202-4c64-a703-b4187bbe92ad","Type":"ContainerDied","Data":"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"}
Sep 10 00:26:50.114592 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.114482    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"
Sep 10 00:26:50.114592 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.114521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b" event={"ID":"9c5e98b3-2202-4c64-a703-b4187bbe92ad","Type":"ContainerDied","Data":"1a57898970b75bf51cfbf876886e0baa05a38e90cfc8421ac893936c469f2064"}
Sep 10 00:26:50.124322 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.124301    2576 scope.go:117] "RemoveContainer" containerID="48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"
Sep 10 00:26:50.124661 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:26:50.124628    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f\": container with ID starting with 48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f not found: ID does not exist" containerID="48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"
Sep 10 00:26:50.124661 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.124651    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f"} err="failed to get container status \"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f\": rpc error: code = NotFound desc = could not find container \"48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f\": container with ID starting with 48e3278b60c0a4453675ef54ee57f062c9660a2757c0c29d19ae475f0b63533f not found: ID does not exist"
Sep 10 00:26:50.124813 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.124668    2576 scope.go:117] "RemoveContainer" containerID="99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"
Sep 10 00:26:50.133124 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.133091    2576 scope.go:117] "RemoveContainer" containerID="99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"
Sep 10 00:26:50.133418 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:26:50.133397    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a\": container with ID starting with 99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a not found: ID does not exist" containerID="99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"
Sep 10 00:26:50.133496 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.133423    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a"} err="failed to get container status \"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a\": rpc error: code = NotFound desc = could not find container \"99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a\": container with ID starting with 99c2b5469c9781f8d83050627e94a5a57a043c42894b40016ab15018eb94b04a not found: ID does not exist"
Sep 10 00:26:50.135171 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.135152    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:26:50.138986 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.138960    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1de9a7-5c795988b6-sskqk"]
Sep 10 00:26:50.148374 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.148352    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:26:50.151843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:50.151812    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1de9a7-87ff8d57c-m4f4b"]
Sep 10 00:26:51.201849 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:51.201816    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27583020-b85b-4e47-8cd0-f1022277a989" path="/var/lib/kubelet/pods/27583020-b85b-4e47-8cd0-f1022277a989/volumes"
Sep 10 00:26:51.202230 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:26:51.202153    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" path="/var/lib/kubelet/pods/9c5e98b3-2202-4c64-a703-b4187bbe92ad/volumes"
Sep 10 00:27:06.600342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:06.600298    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:27:06.600880 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:06.600592    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" podUID="587f16a6-9716-4431-88c2-2f5faa8acc41" containerName="payload-processing" containerID="cri-o://29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208" gracePeriod=30
Sep 10 00:27:06.857109 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:06.857057    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:27:07.018875 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.018847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r74zk\" (UniqueName: \"kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk\") pod \"587f16a6-9716-4431-88c2-2f5faa8acc41\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") "
Sep 10 00:27:07.019034 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.018892    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume\") pod \"587f16a6-9716-4431-88c2-2f5faa8acc41\" (UID: \"587f16a6-9716-4431-88c2-2f5faa8acc41\") "
Sep 10 00:27:07.019278 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.019257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "587f16a6-9716-4431-88c2-2f5faa8acc41" (UID: "587f16a6-9716-4431-88c2-2f5faa8acc41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:27:07.021142 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.021114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk" (OuterVolumeSpecName: "kube-api-access-r74zk") pod "587f16a6-9716-4431-88c2-2f5faa8acc41" (UID: "587f16a6-9716-4431-88c2-2f5faa8acc41"). InnerVolumeSpecName "kube-api-access-r74zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:27:07.119838 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.119778    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74zk\" (UniqueName: \"kubernetes.io/projected/587f16a6-9716-4431-88c2-2f5faa8acc41-kube-api-access-r74zk\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:27:07.119838 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.119806    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/587f16a6-9716-4431-88c2-2f5faa8acc41-plugins-config-volume\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:27:07.208758 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.208726    2576 generic.go:356] "Generic (PLEG): container finished" podID="587f16a6-9716-4431-88c2-2f5faa8acc41" containerID="29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208" exitCode=0
Sep 10 00:27:07.208902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.208770    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"
Sep 10 00:27:07.208902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.208813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" event={"ID":"587f16a6-9716-4431-88c2-2f5faa8acc41","Type":"ContainerDied","Data":"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"}
Sep 10 00:27:07.208902 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.208872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9" event={"ID":"587f16a6-9716-4431-88c2-2f5faa8acc41","Type":"ContainerDied","Data":"e02a88dacd8aea8bdc01ae178e646cb77e077f26bc6bc78f886a424daec46c37"}
Sep 10 00:27:07.209071 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.208901    2576 scope.go:117] "RemoveContainer" containerID="29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"
Sep 10 00:27:07.218773 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.218756    2576 scope.go:117] "RemoveContainer" containerID="29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"
Sep 10 00:27:07.219045 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:27:07.219025    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208\": container with ID starting with 29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208 not found: ID does not exist" containerID="29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"
Sep 10 00:27:07.219089 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.219054    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208"} err="failed to get container status \"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208\": rpc error: code = NotFound desc = could not find container \"29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208\": container with ID starting with 29f2862eed2a31b612ad1360762823183a2b4b9389389f6c7ca44f31f2ec5208 not found: ID does not exist"
Sep 10 00:27:07.225157 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.225136    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:27:07.228167 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:07.228144    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-b1baac-5678c7d86f-ns8b9"]
Sep 10 00:27:09.201886 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:09.201854    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587f16a6-9716-4431-88c2-2f5faa8acc41" path="/var/lib/kubelet/pods/587f16a6-9716-4431-88c2-2f5faa8acc41/volumes"
Sep 10 00:27:47.643678 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:27:47.643642    2576 scope.go:117] "RemoveContainer" containerID="547f52b18b4dc98894eb7ca66a231078961c74f222734b69749514aaa2567fa4"
Sep 10 00:31:35.336817 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.336783    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337277    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27583020-b85b-4e47-8cd0-f1022277a989" containerName="payload-pre-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337296    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="27583020-b85b-4e47-8cd0-f1022277a989" containerName="payload-pre-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337321    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587f16a6-9716-4431-88c2-2f5faa8acc41" containerName="payload-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337331    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="587f16a6-9716-4431-88c2-2f5faa8acc41" containerName="payload-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337341    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1994db70-a1ec-4305-b3ea-ecde6df43d45" containerName="payload-pre-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337349    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1994db70-a1ec-4305-b3ea-ecde6df43d45" containerName="payload-pre-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337362    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" containerName="payload-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337370    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" containerName="payload-processing"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337382    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20629073-fdfc-421c-ac5d-51d7d26018bf" containerName="maas-api"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337392    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="20629073-fdfc-421c-ac5d-51d7d26018bf" containerName="maas-api"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337411    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" containerName="maas-api"
Sep 10 00:31:35.337426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337419    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" containerName="maas-api"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337496    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="20629073-fdfc-421c-ac5d-51d7d26018bf" containerName="maas-api"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337509    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c5e98b3-2202-4c64-a703-b4187bbe92ad" containerName="payload-processing"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337519    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="587f16a6-9716-4431-88c2-2f5faa8acc41" containerName="payload-processing"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337528    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="27583020-b85b-4e47-8cd0-f1022277a989" containerName="payload-pre-processing"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337560    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1994db70-a1ec-4305-b3ea-ecde6df43d45" containerName="payload-pre-processing"
Sep 10 00:31:35.338054 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.337574    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9b1424-37ee-4234-8c4c-bc7ddd4e4688" containerName="maas-api"
Sep 10 00:31:35.340756 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.340735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.343169 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.343149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-616e069e-kserve-self-signed-certs\""
Sep 10 00:31:35.349264 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.348935    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:31:35.413760 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.413725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.413929 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.413764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.413929 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.413840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb6r\" (UniqueName: \"kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.413929 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.413908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.414055 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.413943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.414055 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.414005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515028 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.514987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb6r\" (UniqueName: \"kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515215 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515447 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515563 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515635 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.515673 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.515646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.517620 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.517600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.517802 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.517784    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.521895 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.521873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb6r\" (UniqueName: \"kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r\") pod \"e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") " pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.652408 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.652310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:35.922011 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:35.921984    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:31:35.923583 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:31:35.923550    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod136cbd77_3e6e_44b5_b1eb_28332d4b52a6.slice/crio-f93f1678227f0ca888791aa152f7405e2728091732a1822407a3009f115bfed0 WatchSource:0}: Error finding container f93f1678227f0ca888791aa152f7405e2728091732a1822407a3009f115bfed0: Status 404 returned error can't find the container with id f93f1678227f0ca888791aa152f7405e2728091732a1822407a3009f115bfed0
Sep 10 00:31:36.566425 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:36.566389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" event={"ID":"136cbd77-3e6e-44b5-b1eb-28332d4b52a6","Type":"ContainerStarted","Data":"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"}
Sep 10 00:31:36.566425 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:36.566427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" event={"ID":"136cbd77-3e6e-44b5-b1eb-28332d4b52a6","Type":"ContainerStarted","Data":"f93f1678227f0ca888791aa152f7405e2728091732a1822407a3009f115bfed0"}
Sep 10 00:31:36.583929 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:36.583872    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" podStartSLOduration=1.583857131 podStartE2EDuration="1.583857131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:31:36.581487553 +0000 UTC m=+2569.901863623" watchObservedRunningTime="2026-09-10 00:31:36.583857131 +0000 UTC m=+2569.904233247"
Sep 10 00:31:45.652741 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:45.652706    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:45.653134 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:45.652756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:45.665693 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:45.665670    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:31:46.635150 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:31:46.635120    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:32:29.242327 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:29.242244    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:32:44.262403 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.262360    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" podUID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" containerName="main" containerID="cri-o://8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919" gracePeriod=45
Sep 10 00:32:44.510928 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.510901    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:32:44.641213 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641258    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqb6r\" (UniqueName: \"kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641409 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641692 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641445    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache\") pod \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\" (UID: \"136cbd77-3e6e-44b5-b1eb-28332d4b52a6\") "
Sep 10 00:32:44.641692 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:32:44.641692 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home" (OuterVolumeSpecName: "home") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:32:44.641832 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641694    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:32:44.641832 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641808    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.641832 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641829    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.641968 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.641842    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.643749 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.643706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm" (OuterVolumeSpecName: "dshm") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:32:44.643854 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.643723    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r" (OuterVolumeSpecName: "kube-api-access-bqb6r") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "kube-api-access-bqb6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:32:44.643854 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.643800    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "136cbd77-3e6e-44b5-b1eb-28332d4b52a6" (UID: "136cbd77-3e6e-44b5-b1eb-28332d4b52a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:32:44.742755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.742719    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqb6r\" (UniqueName: \"kubernetes.io/projected/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-kube-api-access-bqb6r\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.742755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.742751    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.742755 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.742762    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cbd77-3e6e-44b5-b1eb-28332d4b52a6-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:32:44.936422 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.936333    2576 generic.go:356] "Generic (PLEG): container finished" podID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" containerID="8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919" exitCode=0
Sep 10 00:32:44.936422 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.936377    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"
Sep 10 00:32:44.936694 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.936418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" event={"ID":"136cbd77-3e6e-44b5-b1eb-28332d4b52a6","Type":"ContainerDied","Data":"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"}
Sep 10 00:32:44.936694 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.936464    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6" event={"ID":"136cbd77-3e6e-44b5-b1eb-28332d4b52a6","Type":"ContainerDied","Data":"f93f1678227f0ca888791aa152f7405e2728091732a1822407a3009f115bfed0"}
Sep 10 00:32:44.936694 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.936484    2576 scope.go:117] "RemoveContainer" containerID="8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"
Sep 10 00:32:44.946656 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.946610    2576 scope.go:117] "RemoveContainer" containerID="8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"
Sep 10 00:32:44.946924 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:32:44.946907    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919\": container with ID starting with 8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919 not found: ID does not exist" containerID="8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"
Sep 10 00:32:44.946967 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.946933    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919"} err="failed to get container status \"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919\": rpc error: code = NotFound desc = could not find container \"8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919\": container with ID starting with 8659db2646a1a56b6d60b17b75811655e1c9d1e4d54605ee42044c2ed9b79919 not found: ID does not exist"
Sep 10 00:32:44.957012 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.956988    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:32:44.960794 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:44.960775    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-616e069e-kserve-7b4fbd5599-shnp6"]
Sep 10 00:32:45.202602 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:45.202511    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" path="/var/lib/kubelet/pods/136cbd77-3e6e-44b5-b1eb-28332d4b52a6/volumes"
Sep 10 00:32:55.546662 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.546624    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"]
Sep 10 00:32:55.547135 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.547116    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" containerName="main"
Sep 10 00:32:55.547194 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.547141    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" containerName="main"
Sep 10 00:32:55.547273 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.547261    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="136cbd77-3e6e-44b5-b1eb-28332d4b52a6" containerName="main"
Sep 10 00:32:55.550658 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.550635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.553043 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.553025    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h47kt\""
Sep 10 00:32:55.558745 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.558724    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"]
Sep 10 00:32:55.643661 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.643622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcqr\" (UniqueName: \"kubernetes.io/projected/552867a2-644d-4bec-801c-5c036b1a7e27-kube-api-access-kkcqr\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.643813 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.643685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-metrics-certs\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.643813 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.643754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552867a2-644d-4bec-801c-5c036b1a7e27-openshift-service-ca\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.643900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.643821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-maas-api-tls\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.744514 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.744481    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-maas-api-tls\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.744697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.744556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcqr\" (UniqueName: \"kubernetes.io/projected/552867a2-644d-4bec-801c-5c036b1a7e27-kube-api-access-kkcqr\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.744697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.744596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-metrics-certs\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.744697 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.744615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552867a2-644d-4bec-801c-5c036b1a7e27-openshift-service-ca\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.745291 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.745268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/552867a2-644d-4bec-801c-5c036b1a7e27-openshift-service-ca\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.747004 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.746979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-maas-api-tls\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.747102 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.747085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/552867a2-644d-4bec-801c-5c036b1a7e27-metrics-certs\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.752183 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.752167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcqr\" (UniqueName: \"kubernetes.io/projected/552867a2-644d-4bec-801c-5c036b1a7e27-kube-api-access-kkcqr\") pod \"maas-api-7895df9f99-8z9hv\" (UID: \"552867a2-644d-4bec-801c-5c036b1a7e27\") " pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:55.862207 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:55.862169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:56.135238 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:56.135206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"]
Sep 10 00:32:56.136967 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:32:56.136942    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552867a2_644d_4bec_801c_5c036b1a7e27.slice/crio-2b4feda5d3c3f92b2c64fa5b69f6c522b1615189897c15a5bffff2a2cda7909a WatchSource:0}: Error finding container 2b4feda5d3c3f92b2c64fa5b69f6c522b1615189897c15a5bffff2a2cda7909a: Status 404 returned error can't find the container with id 2b4feda5d3c3f92b2c64fa5b69f6c522b1615189897c15a5bffff2a2cda7909a
Sep 10 00:32:56.138728 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:56.138710    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:32:57.000385 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:57.000351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv" event={"ID":"552867a2-644d-4bec-801c-5c036b1a7e27","Type":"ContainerStarted","Data":"4b4c4dfcf16d148901d7ff5b627fbb7c412ca4bdd5f516de9830ac1ea829265a"}
Sep 10 00:32:57.000802 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:57.000414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv" event={"ID":"552867a2-644d-4bec-801c-5c036b1a7e27","Type":"ContainerStarted","Data":"2b4feda5d3c3f92b2c64fa5b69f6c522b1615189897c15a5bffff2a2cda7909a"}
Sep 10 00:32:57.000802 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:57.000551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:32:57.018084 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:32:57.018039    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv" podStartSLOduration=1.6574578610000001 podStartE2EDuration="2.018024287s" totalImagesPullingTime="360.566426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:32:57.016411428 +0000 UTC m=+2650.336787494" watchObservedRunningTime="2026-09-10 00:32:57.018024287 +0000 UTC m=+2650.338400409"
Sep 10 00:33:03.011630 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:03.011599    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7895df9f99-8z9hv"
Sep 10 00:33:03.054465 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:03.054432    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:33:03.054712 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:03.054672    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" podUID="76c082a7-0baa-4df9-8431-007d7c0cf50f" containerName="maas-api" containerID="cri-o://f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978" gracePeriod=30
Sep 10 00:33:04.299310 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.299281    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:33:04.316990 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.316959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") pod \"76c082a7-0baa-4df9-8431-007d7c0cf50f\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") "
Sep 10 00:33:04.317231 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.317026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca\") pod \"76c082a7-0baa-4df9-8431-007d7c0cf50f\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") "
Sep 10 00:33:04.317231 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.317061    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") pod \"76c082a7-0baa-4df9-8431-007d7c0cf50f\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") "
Sep 10 00:33:04.317231 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.317097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q45b\" (UniqueName: \"kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b\") pod \"76c082a7-0baa-4df9-8431-007d7c0cf50f\" (UID: \"76c082a7-0baa-4df9-8431-007d7c0cf50f\") "
Sep 10 00:33:04.320106 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.320058    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76c082a7-0baa-4df9-8431-007d7c0cf50f" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:33:04.320464 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.320441    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b" (OuterVolumeSpecName: "kube-api-access-2q45b") pod "76c082a7-0baa-4df9-8431-007d7c0cf50f" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f"). InnerVolumeSpecName "kube-api-access-2q45b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:33:04.320655 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.320634    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76c082a7-0baa-4df9-8431-007d7c0cf50f" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:33:04.320714 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.320626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76c082a7-0baa-4df9-8431-007d7c0cf50f" (UID: "76c082a7-0baa-4df9-8431-007d7c0cf50f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:33:04.418030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.417940    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-metrics-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:33:04.418030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.417971    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c082a7-0baa-4df9-8431-007d7c0cf50f-openshift-service-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:33:04.418030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.417981    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c082a7-0baa-4df9-8431-007d7c0cf50f-maas-api-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:33:04.418030 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:04.417991    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q45b\" (UniqueName: \"kubernetes.io/projected/76c082a7-0baa-4df9-8431-007d7c0cf50f-kube-api-access-2q45b\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 10 00:33:05.045699 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.045662    2576 generic.go:356] "Generic (PLEG): container finished" podID="76c082a7-0baa-4df9-8431-007d7c0cf50f" containerID="f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978" exitCode=0
Sep 10 00:33:05.045881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.045709    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" event={"ID":"76c082a7-0baa-4df9-8431-007d7c0cf50f","Type":"ContainerDied","Data":"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"}
Sep 10 00:33:05.045881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.045754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh" event={"ID":"76c082a7-0baa-4df9-8431-007d7c0cf50f","Type":"ContainerDied","Data":"ed62708d80018138c7738c7a0983f48cc4a810ae07844c6f4a76dff298f4cf32"}
Sep 10 00:33:05.045881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.045776    2576 scope.go:117] "RemoveContainer" containerID="f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"
Sep 10 00:33:05.045881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.045716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"
Sep 10 00:33:05.055568 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.055534    2576 scope.go:117] "RemoveContainer" containerID="f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"
Sep 10 00:33:05.055856 ip-10-0-135-244 kubenswrapper[2576]: E0910 00:33:05.055837    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978\": container with ID starting with f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978 not found: ID does not exist" containerID="f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"
Sep 10 00:33:05.055914 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.055864    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978"} err="failed to get container status \"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978\": rpc error: code = NotFound desc = could not find container \"f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978\": container with ID starting with f48bf4653e37e00d2a568ca72152a90de5f54a04b20e787322e4014c06fae978 not found: ID does not exist"
Sep 10 00:33:05.066358 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.066335    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:33:05.071907 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.071868    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-75b96c447f-mcpkh"]
Sep 10 00:33:05.201588 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:05.201551    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c082a7-0baa-4df9-8431-007d7c0cf50f" path="/var/lib/kubelet/pods/76c082a7-0baa-4df9-8431-007d7c0cf50f/volumes"
Sep 10 00:33:22.479276 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:22.479244    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-76lnm_c253a960-4d9e-4366-8a1a-faec84e55cba/manager/0.log"
Sep 10 00:33:22.596098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:22.596069    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hcvtd_85b7d262-9bda-4a79-a77f-ccbf7c01d7e1/manager/0.log"
Sep 10 00:33:22.844000 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:22.843976    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bbd6h_136ee0c6-0264-4cb8-be45-b97dfbec6894/server/0.log"
Sep 10 00:33:23.076565 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:23.076515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-ccgth_fd83a61d-b2c4-4ca6-afd2-090c5d875198/copy-manifests/0.log"
Sep 10 00:33:23.088740 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:23.088711    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-ccgth_fd83a61d-b2c4-4ca6-afd2-090c5d875198/manager/0.log"
Sep 10 00:33:23.670318 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:23.670282    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7895df9f99-8z9hv_552867a2-644d-4bec-801c-5c036b1a7e27/maas-api/0.log"
Sep 10 00:33:24.100836 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:24.100814    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4rk2g_75f19f7b-943f-4eb5-ba28-8bc244cd30c6/postgres/0.log"
Sep 10 00:33:25.205468 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:25.205444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tkqbd_f2007dc4-5449-46d4-9227-7b87eeef29a9/kuadrant-console-plugin/0.log"
Sep 10 00:33:26.576778 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:26.576745    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:33:27.044601 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:27.044516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rc5ps_0339e05e-ad12-4251-baa3-6c611a21bb7d/main/0.log"
Sep 10 00:33:27.371389 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:27.371356    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wbkkg_af1bf8c7-75d6-4d61-a71e-07ee85ea5b53/main/0.log"
Sep 10 00:33:27.733154 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:27.733076    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dfhkp_2f49d8b6-1615-4958-8027-fb9816d04ca9/main/0.log"
Sep 10 00:33:36.738087 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:36.738059    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2prt6_fc1b81b3-022d-4897-bf77-2c0b3b387446/global-pull-secret-syncer/0.log"
Sep 10 00:33:36.925761 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:36.925729    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pr48t_3f909248-ff01-4fa8-a721-245433a608ef/konnectivity-agent/0.log"
Sep 10 00:33:36.967567 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:36.967530    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-244.ec2.internal_bd5b08fa5b23f3cccc007bda111536ab/haproxy/0.log"
Sep 10 00:33:42.060770 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:42.060696    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-tkqbd_f2007dc4-5449-46d4-9227-7b87eeef29a9/kuadrant-console-plugin/0.log"
Sep 10 00:33:43.851916 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:43.851880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cf6b58cb7-clcrc_ebffc26b-9663-4ffc-a222-f1dc4281e944/metrics-server/0.log"
Sep 10 00:33:43.876185 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:43.876157    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-l6cvz_15282bc1-9142-437f-9c46-69c50ee15bd5/monitoring-plugin/0.log"
Sep 10 00:33:44.030313 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:44.030286    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv4hg_7e592fe9-4e63-4417-a036-85e94f80602a/node-exporter/0.log"
Sep 10 00:33:44.049952 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:44.049931    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv4hg_7e592fe9-4e63-4417-a036-85e94f80602a/kube-rbac-proxy/0.log"
Sep 10 00:33:44.069527 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:44.069508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vv4hg_7e592fe9-4e63-4417-a036-85e94f80602a/init-textfile/0.log"
Sep 10 00:33:47.035955 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.035919    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"]
Sep 10 00:33:47.036411 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.036326    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c082a7-0baa-4df9-8431-007d7c0cf50f" containerName="maas-api"
Sep 10 00:33:47.036411 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.036343    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c082a7-0baa-4df9-8431-007d7c0cf50f" containerName="maas-api"
Sep 10 00:33:47.036515 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.036431    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c082a7-0baa-4df9-8431-007d7c0cf50f" containerName="maas-api"
Sep 10 00:33:47.039490 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.039475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.042117 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.042096    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"openshift-service-ca.crt\""
Sep 10 00:33:47.042220 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.042096    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hczq\"/\"default-dockercfg-r77nj\""
Sep 10 00:33:47.043120 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.043103    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"kube-root-ca.crt\""
Sep 10 00:33:47.049599 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.049575    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"]
Sep 10 00:33:47.197942 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.197910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dmx\" (UniqueName: \"kubernetes.io/projected/be05b244-199e-4105-b1b4-e3cf4055c551-kube-api-access-v4dmx\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.198112 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.197967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-proc\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.198112 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.198042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-podres\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.198112 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.198095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-lib-modules\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.198223 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.198151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-sys\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299412 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-proc\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299412 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-podres\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299412 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-lib-modules\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-sys\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-proc\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dmx\" (UniqueName: \"kubernetes.io/projected/be05b244-199e-4105-b1b4-e3cf4055c551-kube-api-access-v4dmx\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-sys\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299724 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-podres\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.299900 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.299727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be05b244-199e-4105-b1b4-e3cf4055c551-lib-modules\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.306584 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.306566    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"kube-root-ca.crt\""
Sep 10 00:33:47.317270 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.317250    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"openshift-service-ca.crt\""
Sep 10 00:33:47.328602 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.328571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dmx\" (UniqueName: \"kubernetes.io/projected/be05b244-199e-4105-b1b4-e3cf4055c551-kube-api-access-v4dmx\") pod \"perf-node-gather-daemonset-cpv5m\" (UID: \"be05b244-199e-4105-b1b4-e3cf4055c551\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.354592 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.354569    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hczq\"/\"default-dockercfg-r77nj\""
Sep 10 00:33:47.362266 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.362243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:47.643181 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:47.643152    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"]
Sep 10 00:33:47.645025 ip-10-0-135-244 kubenswrapper[2576]: W0910 00:33:47.644985    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe05b244_199e_4105_b1b4_e3cf4055c551.slice/crio-6c95fcc900ffd1c0115f450970a5ab97aff0dd974f47e5aa4cb7ac6ad48fae5b WatchSource:0}: Error finding container 6c95fcc900ffd1c0115f450970a5ab97aff0dd974f47e5aa4cb7ac6ad48fae5b: Status 404 returned error can't find the container with id 6c95fcc900ffd1c0115f450970a5ab97aff0dd974f47e5aa4cb7ac6ad48fae5b
Sep 10 00:33:48.272171 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:48.272137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m" event={"ID":"be05b244-199e-4105-b1b4-e3cf4055c551","Type":"ContainerStarted","Data":"674c33b9a8b003e2e9b9635bfbf0aa7cb7f94e08403ae57fc36410e8f26a2a0d"}
Sep 10 00:33:48.272695 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:48.272188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m" event={"ID":"be05b244-199e-4105-b1b4-e3cf4055c551","Type":"ContainerStarted","Data":"6c95fcc900ffd1c0115f450970a5ab97aff0dd974f47e5aa4cb7ac6ad48fae5b"}
Sep 10 00:33:48.272695 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:48.272225    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:48.288227 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:48.288185    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m" podStartSLOduration=1.28817015 podStartE2EDuration="1.28817015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 00:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:33:48.285847605 +0000 UTC m=+2701.606223670" watchObservedRunningTime="2026-09-10 00:33:48.28817015 +0000 UTC m=+2701.608546214"
Sep 10 00:33:49.033455 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.033420    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jscps_ab711869-1ff0-4780-b836-8fac1eea468b/dns/0.log"
Sep 10 00:33:49.054098 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.054072    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jscps_ab711869-1ff0-4780-b836-8fac1eea468b/kube-rbac-proxy/0.log"
Sep 10 00:33:49.135397 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.135372    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hfzpf_54d90c34-7f98-4289-99f4-32f0ecc6b581/dns-node-resolver/0.log"
Sep 10 00:33:49.654497 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.654466    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29816640-m5z7h_fb5333a2-7d15-4e8d-a087-45f5e2236aa8/image-pruner/0.log"
Sep 10 00:33:49.688823 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.688791    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cd55f54d5-shmzb_407c45f9-a300-4a35-86be-f2497932860d/registry/0.log"
Sep 10 00:33:49.749843 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:49.749814    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n8f6b_3ede7ec3-5176-498b-890d-bc4614e18301/node-ca/0.log"
Sep 10 00:33:51.271859 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:51.271827    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:33:52.190277 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.190249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lbz2f_5d2ce9b4-a71a-4589-b1b6-b70aaa2c90f4/serve-healthcheck-canary/0.log"
Sep 10 00:33:52.626732 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.626704    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-h5tx7_359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad/insights-operator/0.log"
Sep 10 00:33:52.627155 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.627078    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-h5tx7_359a4c57-2c5f-40c5-a4a6-aa955cc6b8ad/insights-operator/1.log"
Sep 10 00:33:52.644426 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.644403    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88sm9_cde60247-5f8b-4be1-874a-5286b0bee0bd/kube-rbac-proxy/0.log"
Sep 10 00:33:52.662881 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.662861    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88sm9_cde60247-5f8b-4be1-874a-5286b0bee0bd/exporter/0.log"
Sep 10 00:33:52.682075 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:52.682055    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-88sm9_cde60247-5f8b-4be1-874a-5286b0bee0bd/extractor/0.log"
Sep 10 00:33:54.288028 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:54.288000    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-cpv5m"
Sep 10 00:33:54.805170 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:54.805144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-76lnm_c253a960-4d9e-4366-8a1a-faec84e55cba/manager/0.log"
Sep 10 00:33:55.084819 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:55.084785    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hcvtd_85b7d262-9bda-4a79-a77f-ccbf7c01d7e1/manager/0.log"
Sep 10 00:33:55.162904 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:55.162874    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bbd6h_136ee0c6-0264-4cb8-be45-b97dfbec6894/server/0.log"
Sep 10 00:33:55.344996 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:55.344910    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-ccgth_fd83a61d-b2c4-4ca6-afd2-090c5d875198/manager/0.log"
Sep 10 00:33:55.359504 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:55.359479    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-ccgth_fd83a61d-b2c4-4ca6-afd2-090c5d875198/copy-manifests/0.log"
Sep 10 00:33:56.561977 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:56.561946    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5895cdb77f-pn49w_d9ca9e1b-ebe6-4460-90b2-536f5317901e/manager/0.log"
Sep 10 00:33:57.457669 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:57.457590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:33:59.682354 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:33:59.682315    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7tq89_a07d06a7-aca4-4ae9-912f-5c5727565d09/payload-processing/0.log"
Sep 10 00:34:03.087280 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.087249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/kube-multus-additional-cni-plugins/0.log"
Sep 10 00:34:03.105033 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.105006    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/egress-router-binary-copy/0.log"
Sep 10 00:34:03.122725 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.122705    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/cni-plugins/0.log"
Sep 10 00:34:03.140771 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.140746    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/bond-cni-plugin/0.log"
Sep 10 00:34:03.166176 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.166156    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/routeoverride-cni/0.log"
Sep 10 00:34:03.184342 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.184324    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/whereabouts-cni-bincopy/0.log"
Sep 10 00:34:03.205239 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.205179    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bwtfb_6bbacd63-8d59-408f-9ca0-b9b3b8993815/whereabouts-cni/0.log"
Sep 10 00:34:03.518873 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.518804    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z5qmv_63c61f47-fa07-4006-abdb-e98e855f96a0/kube-multus/0.log"
Sep 10 00:34:03.614059 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.614030    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ttqwg_7698041c-72bd-4509-840e-9add393b3bff/network-metrics-daemon/0.log"
Sep 10 00:34:03.629294 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:03.629272    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ttqwg_7698041c-72bd-4509-840e-9add393b3bff/kube-rbac-proxy/0.log"
Sep 10 00:34:04.615662 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.615627    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/ovn-controller/0.log"
Sep 10 00:34:04.629512 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.629485    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/ovn-acl-logging/0.log"
Sep 10 00:34:04.652473 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.652451    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/kube-rbac-proxy-node/0.log"
Sep 10 00:34:04.671100 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.671083    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 00:34:04.686055 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.686039    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/northd/0.log"
Sep 10 00:34:04.703321 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.703307    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/nbdb/0.log"
Sep 10 00:34:04.721279 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.721256    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/sbdb/0.log"
Sep 10 00:34:04.779008 ip-10-0-135-244 kubenswrapper[2576]: I0910 00:34:04.778982    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st67z_402bf08e-c18b-4a14-ace4-6c259ab7aa8c/ovnkube-controller/0.log"