Sep 15 21:14:01.978460 ip-10-0-135-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 21:14:01.978472 ip-10-0-135-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 21:14:01.978481 ip-10-0-135-105 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 21:14:01.978692 ip-10-0-135-105 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 21:14:12.163858 ip-10-0-135-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 21:14:12.163873 ip-10-0-135-105 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c6cbba5eef65426cab155e04abe2f14b --
Sep 15 21:16:11.495022 ip-10-0-135-105 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:16:11.849290 ip-10-0-135-105 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:16:11.849290 ip-10-0-135-105 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:16:11.849290 ip-10-0-135-105 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:16:11.849290 ip-10-0-135-105 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:16:11.849290 ip-10-0-135-105 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:16:11.851959 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.851859    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:16:11.856253 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856236    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.856253 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856252    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856256    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856261    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856265    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856268    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856271    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856274    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856277    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856280    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856282    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856285    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856295    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856298    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856300    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856303    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856305    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856308    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856312    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856314    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856317    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.856321 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856319    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856322    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856325    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856328    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856331    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856334    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856336    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856339    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856342    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856345    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856347    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856350    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856353    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856355    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856358    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856360    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856363    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856365    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856368    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856371    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.856815 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856373    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856376    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856378    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856381    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856384    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856386    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856389    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856391    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856394    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856396    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856399    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856401    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856404    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856406    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856411    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856414    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856416    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856420    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856423    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856425    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.857299 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856428    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856433    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856436    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856439    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856441    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856444    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856446    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856449    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856452    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856454    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856456    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856460    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856462    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856465    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856467    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856472    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856474    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856477    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856479    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.857832 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856481    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856484    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856487    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856489    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856492    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856494    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856498    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856500    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856503    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856507    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856511    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856514    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856516    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856520    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856524    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856527    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856529    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.856532    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858011    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.858314 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858018    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858021    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858024    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858027    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858031    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858033    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858037    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858039    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858042    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858045    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858048    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858051    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858053    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858056    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858059    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858061    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858064    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858066    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858069    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.858788 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858072    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858075    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858077    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858080    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858083    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858086    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858088    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858091    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858093    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858096    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858099    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858102    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858104    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858107    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858109    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858111    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858114    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858116    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858121    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858124    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.859271 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858128    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858130    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858133    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858137    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858140    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858144    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858148    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858151    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858153    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858156    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858158    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858161    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858164    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858167    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858170    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858172    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858174    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858177    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858179    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858182    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.859768 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858185    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858187    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858190    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858193    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858197    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858199    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858202    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858204    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858207    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858209    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858212    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858214    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858217    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858221    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858226    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858229    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858232    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858234    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858237    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858239    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.860268 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858242    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858245    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858248    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858250    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858253    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858257    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858261    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858264    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858267    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858270    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858272    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858275    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858278    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858281    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858283    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858286    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858288    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858291    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858293    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.860766 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858650    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858654    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858658    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858660    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858663    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858666    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858669    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858674    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858676    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858679    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858682    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858685    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858688    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858691    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858693    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858696    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858698    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858701    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858704    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858707    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.861226 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858710    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858712    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858730    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858732    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858735    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858738    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858743    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858747    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858750    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858753    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858756    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858760    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858764    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858767    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858769    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858772    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858774    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858777    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858779    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.861710 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858784    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858787    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858789    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858792    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858795    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858799    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858802    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858804    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858807    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858809    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858813    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858815    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858818    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858820    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858823    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858825    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858830    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858834    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.862181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858838    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858841    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858843    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858846    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858849    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858852    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858855    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858858    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858861    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858864    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858867    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858869    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858872    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858876    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858879    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858882    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858884    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858887    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858890    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858892    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.862679 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858895    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858897    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858900    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858903    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858905    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858908    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858910    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858914    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858916    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858919    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858921    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858924    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858927    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858929    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858932    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858934    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858936    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858939    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858941    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858944    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.863269 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.858947    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859013    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859047    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859054    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859059    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859063    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859066    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859071    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859076    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859079    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859082    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859085    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859088    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859092    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859095    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859098    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859101    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859104    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859108    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859112    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859115    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859118    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859121    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859125    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:16:11.863768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859128    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859131    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859134    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859138    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859141    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859144    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859147    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859150    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859153    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859157    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859160    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859163    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859166    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859169    2578 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859172    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859177    2578 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859181    2578 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859184    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859187    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859190    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859194    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859197    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859200    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859203    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859206    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:16:11.864398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859209    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859212    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859214    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859217    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859220    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859223    2578 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859227    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859230    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859233    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859236    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859240    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859243    2578 flags.go:64] FLAG: --help="false"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859246    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-135-105.ec2.internal"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859249    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859252    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859255    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859259    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859262    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859265    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859268    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859271    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859274    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859278    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859281    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:16:11.865047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859285    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859288    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859290    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859294    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859296    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859299    2578 flags.go:64] FLAG: --lock-file=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859302    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859305    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859308    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859313    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859316    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859319    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859322    2578 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859325    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859328    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859331    2578 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859334    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859339    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859342    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859346    2578 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859349    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859352    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859355    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859358    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859361    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:16:11.865647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859364    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859367    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859375    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859378    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859381    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859384    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859387    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859392    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859396    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859399    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859402    2578 flags.go:64] FLAG: --port="10250"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859406    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859409    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f593574a9659084b"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859413    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859417    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859420    2578 flags.go:64] FLAG: --register-node="true"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859422    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859426    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859429    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859432    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859435    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859438    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859441    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859444    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859447    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:16:11.866325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859450    2578 flags.go:64] FLAG: --runonce="false"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859453    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859456    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859459    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859462    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859465    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859468    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859471    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859474    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859478    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859481    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859484    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859487    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859489    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859492    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859495    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859502    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859505    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859508    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859512    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859516    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859519    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859522    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859525    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859528    2578 flags.go:64] FLAG: --v="2"
Sep 15 21:16:11.866968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859532    2578 flags.go:64] FLAG: --version="false"
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859536    2578 flags.go:64] FLAG: --vmodule=""
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859540    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.859544    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859675    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859678    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859681    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859685    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859687    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859690    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859693    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859696    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859699    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859702    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859704    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859707    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859712    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859728    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859730    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859733    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859735    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.867957 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859738    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859741    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859743    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859747    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859749    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859752    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859757    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859759    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859762    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859765    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859767    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859770    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859773    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859776    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859778    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859783    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859787    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859791    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859796    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859799    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.868634 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859801    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859804    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859807    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859809    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859812    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859815    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859817    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859822    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859824    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859827    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859830    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859834    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859838    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859841    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859844    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859847    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859850    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859853    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859857    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859860    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.869152 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859863    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859865    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859868    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859871    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859873    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859876    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859878    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859881    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859884    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859886    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859889    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859892    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859895    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859898    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859900    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859903    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859905    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859908    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859912    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.869651 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859917    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859920    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859924    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859927    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859931    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859934    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859936    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859939    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859942    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859944    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859953    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859958    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859961    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859963    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859966    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859969    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859971    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859974    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859976    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.870135 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859979    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859982    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859984    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.859987    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.860568    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.868794    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.868810    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.869263    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.869270    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869327    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869333    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869337    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869341    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869344    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.870636 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869346    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869349    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869353    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869355    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869358    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869362    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869366    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869370    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869373    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869375    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869378    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869381    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869384    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869387    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869390    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869392    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869395    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869397    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869400    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869402    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.871113 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869405    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869408    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869411    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869414    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869418    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869421    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869423    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869426    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869428    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869432    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869434    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869437    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869440    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869443    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869446    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869448    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869451    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869453    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869456    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.871594 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869458    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869461    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869464    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869466    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869469    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869471    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869474    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869477    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869480    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869482    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869485    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869487    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869490    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869492    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869495    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869497    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869500    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869504    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869507    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869509    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.872077 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869512    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869515    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869517    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869520    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869523    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869525    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869528    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869531    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869533    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869536    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869538    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869541    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869544    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869546    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869549    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869552    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869555    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869558    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869560    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869563    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.872615 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869565    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869568    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869571    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869573    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869576    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869578    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869581    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869584    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869586    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869589    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869592    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869596    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869599    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869602    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869606    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.873125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.869611    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869739    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869746    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869751    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869754    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869757    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869762    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869765    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869768    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869771    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869774    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869777    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869780    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869783    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869786    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869789    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869791    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869794    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869797    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:16:11.873587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869799    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869802    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869804    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869807    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869810    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869812    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869815    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869817    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869820    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869823    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869825    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869828    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869830    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869833    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869836    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869838    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869841    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869843    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869846    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:16:11.874102 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869849    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869851    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869854    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869857    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869859    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869862    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869864    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869867    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869871    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869873    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869876    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869879    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869881    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869884    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869887    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869889    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869892    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869894    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869897    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869899    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:16:11.874970 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869902    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869905    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869909    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869912    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869915    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869918    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869921    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869924    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869927    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869930    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869933    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869935    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869938    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869940    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869943    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869946    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869948    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869951    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869953    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:16:11.875546 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869956    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869958    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869962    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869965    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869967    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869970    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869972    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869975    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869978    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869980    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869983    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869985    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869988    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869990    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869993    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869995    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.869998    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870001    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870003    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870006    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:16:11.876088 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870009    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870011    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:11.870013    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.870019    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.870119    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.872346    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.873149    2578 server.go:1019] "Starting client certificate rotation"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.873262    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.873304    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.875402    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:16:11.876656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.876605    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:16:11.884446 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.884427    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:16:11.887283 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.887266    2578 log.go:25] "Validated CRI v1 image API"
Sep 15 21:16:11.888244 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.888218    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:16:11.893058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.893038    2578 fs.go:135] Filesystem UUIDs: map[2cc1fcbe-65e3-44cb-8065-6cdc17a567a8:/dev/nvme0n1p3 724c1df6-23b3-4f71-89be-b5ed7d8a9936:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 15 21:16:11.893129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.893057    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:16:11.900415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.900285    2578 manager.go:216] Machine: {Timestamp:2026-09-15 21:16:11.899356248 +0000 UTC m=+0.300329490 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100573 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29d03583ed70e7c72828482f71f5c2 SystemUUID:ec29d035-83ed-70e7-c728-28482f71f5c2 BootID:c6cbba5e-ef65-426c-ab15-5e04abe2f14b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:c2:77:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:c2:77:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:18:8f:a4:5a:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:16:11.900415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.900401    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:16:11.900586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.900491    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:16:11.902267 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.902248    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:16:11.902975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.902951    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:16:11.903118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.902978    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:16:11.903748 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.903736    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:16:11.903785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.903750    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:16:11.903785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.903758    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:16:11.904497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.904487    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:16:11.905634 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.905624    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:16:11.908726 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908702    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:16:11.908841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908829    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:16:11.908877 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908861    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:16:11.908877 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908871    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:16:11.908934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908882    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:16:11.908934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908892    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:16:11.908934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.908900    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:16:11.909307 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.909296    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:16:11.909350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.909316    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:16:11.911344 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.911323    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:16:11.913961 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.913945    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:16:11.914036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.913989    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:16:11.915615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915599    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:16:11.915656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915629    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:16:11.915656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915638    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:16:11.915656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915644    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:16:11.915656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915653    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915659    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915672    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915678    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915686    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915691    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915700    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:16:11.915789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.915705    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:16:11.917130 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.917119    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:16:11.917185 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.917131    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:16:11.918401 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.918373    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:16:11.918401 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.918390    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:16:11.920200 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.920185    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 21:16:11.921230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.921217    2578 server.go:1268] "Started kubelet"
Sep 15 21:16:11.921391 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.921323    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:16:11.921475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.921424    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:16:11.921475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.921353    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:16:11.922290 ip-10-0-135-105 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:16:11.923098 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.923082    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:16:11.926934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.926865    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:16:11.928628 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.928608    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:16:11.932316 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.932297    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:16:11.932870 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.931942    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-105.ec2.internal.18d59b1c18728003  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-105.ec2.internal,UID:ip-10-0-135-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-105.ec2.internal,},FirstTimestamp:2026-09-15 21:16:11.921186819 +0000 UTC m=+0.322160055,LastTimestamp:2026-09-15 21:16:11.921186819 +0000 UTC m=+0.322160055,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-105.ec2.internal,}"
Sep 15 21:16:11.932990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.932883    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:16:11.933048 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.933019    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:11.933684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933570    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:16:11.933684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933571    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:16:11.933684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933598    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:16:11.933859 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933706    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:16:11.933859 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933729    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:16:11.933962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933924    2578 factory.go:55] Registering systemd factory
Sep 15 21:16:11.933962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.933936    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:16:11.934173 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934150    2578 factory.go:153] Registering CRI-O factory
Sep 15 21:16:11.934173 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934165    2578 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:16:11.934243 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934231    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:16:11.934282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934250    2578 factory.go:103] Registering Raw factory
Sep 15 21:16:11.934282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934266    2578 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:16:11.934894 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.934878    2578 manager.go:349] Starting recovery of all containers
Sep 15 21:16:11.936532 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.936498    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 21:16:11.937876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.937848    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x25fm"
Sep 15 21:16:11.943058 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.942875    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 21:16:11.946641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.946616    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x25fm"
Sep 15 21:16:11.949350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.949334    2578 manager.go:354] Recovery completed
Sep 15 21:16:11.954174 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.954157    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:11.955913 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.955891    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:11.956040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.955937    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:11.956040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.955964    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:11.956502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.956488    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:16:11.956502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.956498    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:11.956626 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.956499    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:16:11.956626 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.956537    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:16:11.957240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957224    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:11.957348 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957255    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:11.957348 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957269    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:11.957632 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957617    2578 policy_none.go:49] "None policy: Start"
Sep 15 21:16:11.957632 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957634    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:16:11.957749 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.957644    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:16:11.958415 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.958321    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-105.ec2.internal.18d59b1c1a847933  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-105.ec2.internal,UID:ip-10-0-135-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-105.ec2.internal,},FirstTimestamp:2026-09-15 21:16:11.955919155 +0000 UTC m=+0.356892395,LastTimestamp:2026-09-15 21:16:11.955919155 +0000 UTC m=+0.356892395,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-105.ec2.internal,}"
Sep 15 21:16:11.958603 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.958591    2578 policy_none.go:47] "Start"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.983279    2578 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.983314    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.983328    2578 server.go:87] "Starting device plugin registration server"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.983747    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.983758    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.983920    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.984005    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:11.984015    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.984555    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:16:11.996516 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:11.984604    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.067443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.067395    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:16:12.068652 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.068634    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:16:12.068775 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.068693    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:16:12.068775 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.068737    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:16:12.068880 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.068787    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:16:12.071231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.071212    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:12.084330 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.084315    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:12.085196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.085178    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:12.085277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.085210    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:12.085277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.085224    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:12.085277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.085253    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.094198 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.094182    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.094292 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.094208    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-105.ec2.internal\": node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.114073 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.113996    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.169340 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.169305    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"]
Sep 15 21:16:12.169498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.169376    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:12.170944 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.170925    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:12.171038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.170964    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:12.171038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.170974    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:12.172056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172044    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:12.172212 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.172269 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172227    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:12.172821 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172798    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:12.172821 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172804    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:12.172964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172828    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:12.172964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172842    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:12.172964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172842    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:12.172964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.172933    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:12.173921 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.173905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.173993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.173932    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:16:12.174599 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.174586    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:16:12.174691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.174613    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:16:12.174691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.174627    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:16:12.198309 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.198288    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-105.ec2.internal\" not found" node="ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.202533 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.202516    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-105.ec2.internal\" not found" node="ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.214241 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.214218    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.236288 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.236261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.236397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.236289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54b69627995a64131f3948c3f8331567-config\") pod \"kube-apiserver-proxy-ip-10-0-135-105.ec2.internal\" (UID: \"54b69627995a64131f3948c3f8331567\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.236397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.236309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.314884 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.314857    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.336979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.336950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.337091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.336992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.337091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.337010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54b69627995a64131f3948c3f8331567-config\") pod \"kube-apiserver-proxy-ip-10-0-135-105.ec2.internal\" (UID: \"54b69627995a64131f3948c3f8331567\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.337091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.336954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.337091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.337062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b90c546182f4af06bbc93616788e0ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal\" (UID: \"7b90c546182f4af06bbc93616788e0ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.337091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.337043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/54b69627995a64131f3948c3f8331567-config\") pod \"kube-apiserver-proxy-ip-10-0-135-105.ec2.internal\" (UID: \"54b69627995a64131f3948c3f8331567\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.415038 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.414960    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.502388 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.502356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.506825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.506671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:12.516095 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.516077    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.622041 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.621990    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.728618 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.728538    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.830379 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.830347    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.840276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.840247    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:12.873044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.873012    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:16:12.873468 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.873160    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:16:12.873468 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.873191    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:16:12.930647 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:12.930618    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:12.951663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.951617    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:11:12 +0000 UTC" deadline="2028-05-03 07:29:34.740947554 +0000 UTC"
Sep 15 21:16:12.951663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:12.951660    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14290h13m21.789291535s"
Sep 15 21:16:13.031406 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.031324    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:13.102266 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:13.102229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b90c546182f4af06bbc93616788e0ad.slice/crio-5dd8b413ff8924ebc88d10582a20517a84f5e886837ae11325617ecfb568d915 WatchSource:0}: Error finding container 5dd8b413ff8924ebc88d10582a20517a84f5e886837ae11325617ecfb568d915: Status 404 returned error can't find the container with id 5dd8b413ff8924ebc88d10582a20517a84f5e886837ae11325617ecfb568d915
Sep 15 21:16:13.102494 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:13.102472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b69627995a64131f3948c3f8331567.slice/crio-83b2d3d87995be6159989541b6a45a40eccf14cfee8e4bdf5485d49734c35465 WatchSource:0}: Error finding container 83b2d3d87995be6159989541b6a45a40eccf14cfee8e4bdf5485d49734c35465: Status 404 returned error can't find the container with id 83b2d3d87995be6159989541b6a45a40eccf14cfee8e4bdf5485d49734c35465
Sep 15 21:16:13.109232 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.109212    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:16:13.131792 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.131762    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:13.238439 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.238401    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:13.339076 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.338996    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:13.439204 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.439167    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-105.ec2.internal\" not found"
Sep 15 21:16:13.472702 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.472673    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:13.502311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.502283    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:13.533211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.533181    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:13.545181 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.545152    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:16:13.546218 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.546197    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal"
Sep 15 21:16:13.555981 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.555955    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:16:13.708768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.708667    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:13.911575 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.911543    2578 apiserver.go:52] "Watching apiserver"
Sep 15 21:16:13.916481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.916451    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:16:13.917674 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.917645    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-vctmx","openshift-ovn-kubernetes/ovnkube-node-75t4g","kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf","openshift-cluster-node-tuning-operator/tuned-xkkcg","openshift-image-registry/node-ca-v4fxj","openshift-multus/network-metrics-daemon-krh4j","openshift-network-diagnostics/network-check-target-pd7zt","kube-system/global-pull-secret-syncer-z4tbr","kube-system/konnectivity-agent-wltw9","openshift-dns/node-resolver-xwltq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-9ngcv","openshift-multus/multus-h468b"]
Sep 15 21:16:13.919239 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.919213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:13.919389 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.919299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:13.920727 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.920708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.922047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.922025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.922879 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.922667    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.922992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.922885    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.922992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.922969    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:16:13.924126 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.923210    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbrgb\""
Sep 15 21:16:13.924126 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.923416    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:16:13.924126 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.923612    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.924953 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.924766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.925354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:16:13.925443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925400    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hfzd\""
Sep 15 21:16:13.925504 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:16:13.925563 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.925637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:16:13.925861 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925834    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.925978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.925877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8vzmv\""
Sep 15 21:16:13.926539 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.926316    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.926622 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.926581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:13.927773 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.927622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:13.928220 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.927947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:13.930457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.929321    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.930457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.929604    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6zsv\""
Sep 15 21:16:13.930457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.930176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.930457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.930393    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:16:13.930812 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.930794    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.930941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.930794    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lptwv\""
Sep 15 21:16:13.931457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.931415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:13.933659 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.933615    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:16:13.933709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.933678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.934006 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.933955    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.934538 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.934410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:16:13.934538 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.934412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.934756 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.934736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhdv7\""
Sep 15 21:16:13.936407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.935319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:13.937355 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.936906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.938976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939009    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2h9vl\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939303    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnkn6\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939754    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-czth7\""
Sep 15 21:16:13.939978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.939816    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:16:13.940620 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.940101    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:16:13.941340 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.941319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:13.941442 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:13.941392    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948274    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjfk\" (UniqueName: \"kubernetes.io/projected/1d421be5-c692-4b4e-9510-ad6741f05228-kube-api-access-2jjfk\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-conf-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-multus-certs\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.948640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv59\" (UniqueName: \"kubernetes.io/projected/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-kube-api-access-9fv59\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-bin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-systemd-units\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98vw\" (UniqueName: \"kubernetes.io/projected/5c74e18b-055c-4cbe-87a4-66683c7dbc19-kube-api-access-c98vw\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-etc-kubernetes\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-env-overrides\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-sys-fs\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b312a59f-32c2-4483-8147-28b5e2f0df15-konnectivity-ca\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.948984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlq7g\" (UniqueName: \"kubernetes.io/projected/d46f5857-c346-4595-a2a4-68ed885c4ce3-kube-api-access-jlq7g\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-log-socket\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-script-lib\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-daemon-config\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-tmp\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-netd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.949190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-dbus\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbsg9\" (UniqueName: \"kubernetes.io/projected/34463d15-8b3b-489d-b214-ba7573a8f230-kube-api-access-jbsg9\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-system-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-netns\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-kubelet\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-kubernetes\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-registration-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-var-lib-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-ovn\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-bin\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852sp\" (UniqueName: \"kubernetes.io/projected/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kube-api-access-852sp\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d421be5-c692-4b4e-9510-ad6741f05228-serviceca\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:13.950067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-modprobe-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-slash\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-node-log\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-var-lib-kubelet\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-tuned\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-config\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-conf\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.949993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-host\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-iptables-alerter-script\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-host-slash\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-socket-dir-parent\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950110    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-multus\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-run\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:13.950880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cni-binary-copy\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b312a59f-32c2-4483-8147-28b5e2f0df15-agent-certs\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-cnibin\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cnibin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-hostroot\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-sys\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-lib-modules\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-kubelet\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-socket-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysconfig\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-systemd\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrnt\" (UniqueName: \"kubernetes.io/projected/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-kube-api-access-pmrnt\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.950935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-systemd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-device-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d421be5-c692-4b4e-9510-ad6741f05228-host\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:13.951699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-k8s-cni-cncf-io\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:13.952670 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-netns\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.952670 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-etc-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.952670 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjdsr\" (UniqueName: \"kubernetes.io/projected/51c05d54-7e85-40e1-8b84-94efb601d3e3-kube-api-access-sjdsr\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:13.952670 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.951212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-kubelet-config\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:13.969956 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.969927    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvgvr"
Sep 15 21:16:13.979231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:13.979206    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvgvr"
Sep 15 21:16:14.035204 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.035128    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:16:14.052205 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjdsr\" (UniqueName: \"kubernetes.io/projected/51c05d54-7e85-40e1-8b84-94efb601d3e3-kube-api-access-sjdsr\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-kubelet-config\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-hosts-file\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-tmp-dir\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-kubelet-config\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.052382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjfk\" (UniqueName: \"kubernetes.io/projected/1d421be5-c692-4b4e-9510-ad6741f05228-kube-api-access-2jjfk\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-conf-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-multus-certs\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.052470    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv59\" (UniqueName: \"kubernetes.io/projected/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-kube-api-access-9fv59\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-bin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-systemd-units\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98vw\" (UniqueName: \"kubernetes.io/projected/5c74e18b-055c-4cbe-87a4-66683c7dbc19-kube-api-access-c98vw\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-etc-kubernetes\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-env-overrides\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-bin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-sys-fs\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b312a59f-32c2-4483-8147-28b5e2f0df15-konnectivity-ca\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-conf-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.052831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-multus-certs\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.052811    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:14.552725991 +0000 UTC m=+2.953699220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-etc-kubernetes\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-sys-fs\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlq7g\" (UniqueName: \"kubernetes.io/projected/d46f5857-c346-4595-a2a4-68ed885c4ce3-kube-api-access-jlq7g\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-etc-selinux\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-log-socket\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-script-lib\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-systemd-units\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-log-socket\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.052992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-daemon-config\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-tmp\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-netd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.053813 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-dbus\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jqn\" (UniqueName: \"kubernetes.io/projected/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-kube-api-access-k9jqn\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbsg9\" (UniqueName: \"kubernetes.io/projected/34463d15-8b3b-489d-b214-ba7573a8f230-kube-api-access-jbsg9\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-system-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-netns\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-kubelet\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-netd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-kubernetes\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-daemon-config\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b312a59f-32c2-4483-8147-28b5e2f0df15-konnectivity-ca\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-script-lib\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-registration-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.054706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053706    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-dbus\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-system-cni-dir\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-kubernetes\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-kubelet\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-var-lib-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-netns\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-ovn\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-var-lib-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-bin\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-ovn-kubernetes\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-cni-bin\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-ovn\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.053990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-registration-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852sp\" (UniqueName: \"kubernetes.io/projected/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kube-api-access-852sp\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d421be5-c692-4b4e-9510-ad6741f05228-serviceca\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-modprobe-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.055602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-slash\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-node-log\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-modprobe-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054203    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-var-lib-kubelet\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-env-overrides\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-slash\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-var-lib-kubelet\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-node-log\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-tuned\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-config\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-conf\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-host\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-system-cni-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-host\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.056508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d421be5-c692-4b4e-9510-ad6741f05228-serviceca\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-iptables-alerter-script\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-host-slash\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-socket-dir-parent\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-multus\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-run\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-d\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysctl-conf\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.054735    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cni-binary-copy\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b312a59f-32c2-4483-8147-28b5e2f0df15-agent-certs\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.054815    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:14.554798288 +0000 UTC m=+2.955771519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-cnibin\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-multus-socket-dir-parent\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cnibin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.057407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-var-lib-cni-multus\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-hostroot\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-sys\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-hostroot\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-lib-modules\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.054977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-kubelet\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-socket-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-sys\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysconfig\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-sysconfig\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-systemd\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrnt\" (UniqueName: \"kubernetes.io/projected/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-kube-api-access-pmrnt\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-systemd\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-systemd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-device-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-run\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.058297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d421be5-c692-4b4e-9510-ad6741f05228-host\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-k8s-cni-cncf-io\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-netns\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovnkube-config\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-etc-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-host-slash\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-etc-openvswitch\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d46f5857-c346-4595-a2a4-68ed885c4ce3-lib-modules\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-kubelet\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-socket-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34463d15-8b3b-489d-b214-ba7573a8f230-cnibin\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-device-dir\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-iptables-alerter-script\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cni-binary-copy\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-host-run-k8s-cni-cncf-io\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d421be5-c692-4b4e-9510-ad6741f05228-host\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-host-run-netns\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51c05d54-7e85-40e1-8b84-94efb601d3e3-run-systemd\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059101 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.055928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c74e18b-055c-4cbe-87a4-66683c7dbc19-cnibin\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.059725 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.056007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34463d15-8b3b-489d-b214-ba7573a8f230-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.059725 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.058957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-tmp\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.059725 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.059641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51c05d54-7e85-40e1-8b84-94efb601d3e3-ovn-node-metrics-cert\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.059917 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.059892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d46f5857-c346-4595-a2a4-68ed885c4ce3-etc-tuned\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.061501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.061470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b312a59f-32c2-4483-8147-28b5e2f0df15-agent-certs\") pod \"konnectivity-agent-wltw9\" (UID: \"b312a59f-32c2-4483-8147-28b5e2f0df15\") " pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:14.066899 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.066602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv59\" (UniqueName: \"kubernetes.io/projected/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-kube-api-access-9fv59\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:14.067456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.067432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjfk\" (UniqueName: \"kubernetes.io/projected/1d421be5-c692-4b4e-9510-ad6741f05228-kube-api-access-2jjfk\") pod \"node-ca-v4fxj\" (UID: \"1d421be5-c692-4b4e-9510-ad6741f05228\") " pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.067573 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.067474    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852sp\" (UniqueName: \"kubernetes.io/projected/2c054e39-ac62-4c9e-8ed6-8fcc550f4d58-kube-api-access-852sp\") pod \"aws-ebs-csi-driver-node-gtxqf\" (UID: \"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.067573 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.067502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98vw\" (UniqueName: \"kubernetes.io/projected/5c74e18b-055c-4cbe-87a4-66683c7dbc19-kube-api-access-c98vw\") pod \"multus-h468b\" (UID: \"5c74e18b-055c-4cbe-87a4-66683c7dbc19\") " pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.068729 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.068686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrnt\" (UniqueName: \"kubernetes.io/projected/cb5e4f3e-9cbb-40d5-8cef-a3252987d135-kube-api-access-pmrnt\") pod \"iptables-alerter-vctmx\" (UID: \"cb5e4f3e-9cbb-40d5-8cef-a3252987d135\") " pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.068876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.068853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlq7g\" (UniqueName: \"kubernetes.io/projected/d46f5857-c346-4595-a2a4-68ed885c4ce3-kube-api-access-jlq7g\") pod \"tuned-xkkcg\" (UID: \"d46f5857-c346-4595-a2a4-68ed885c4ce3\") " pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.069513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.069489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjdsr\" (UniqueName: \"kubernetes.io/projected/51c05d54-7e85-40e1-8b84-94efb601d3e3-kube-api-access-sjdsr\") pod \"ovnkube-node-75t4g\" (UID: \"51c05d54-7e85-40e1-8b84-94efb601d3e3\") " pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.069698 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.069683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbsg9\" (UniqueName: \"kubernetes.io/projected/34463d15-8b3b-489d-b214-ba7573a8f230-kube-api-access-jbsg9\") pod \"multus-additional-cni-plugins-9ngcv\" (UID: \"34463d15-8b3b-489d-b214-ba7573a8f230\") " pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.074077 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.074001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal" event={"ID":"54b69627995a64131f3948c3f8331567","Type":"ContainerStarted","Data":"83b2d3d87995be6159989541b6a45a40eccf14cfee8e4bdf5485d49734c35465"}
Sep 15 21:16:14.075312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.075289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal" event={"ID":"7b90c546182f4af06bbc93616788e0ad","Type":"ContainerStarted","Data":"5dd8b413ff8924ebc88d10582a20517a84f5e886837ae11325617ecfb568d915"}
Sep 15 21:16:14.155675 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.155641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-hosts-file\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.155871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.155687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-tmp-dir\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.155871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.155736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jqn\" (UniqueName: \"kubernetes.io/projected/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-kube-api-access-k9jqn\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.155871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.155798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:14.155871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.155813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-hosts-file\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.156200 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.156173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-tmp-dir\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.168530 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.168502    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:14.168530 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.168530    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:14.168739 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.168541    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:14.168739 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.168614    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:14.668597671 +0000 UTC m=+3.069570897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:14.171166 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.171141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jqn\" (UniqueName: \"kubernetes.io/projected/a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2-kube-api-access-k9jqn\") pod \"node-resolver-xwltq\" (UID: \"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2\") " pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.238927 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.238842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ngcv"
Sep 15 21:16:14.252957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.252929    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h468b"
Sep 15 21:16:14.263099 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.263074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf"
Sep 15 21:16:14.270786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.270766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkkcg"
Sep 15 21:16:14.279483 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.279459    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v4fxj"
Sep 15 21:16:14.290210 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.290181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vctmx"
Sep 15 21:16:14.299142 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.299124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:14.306055 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.306031    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:14.312825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.312803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xwltq"
Sep 15 21:16:14.559832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.559744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:14.559832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.559813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:14.560060 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.559921    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:14.560060 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.559962    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:14.560060 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.560008    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:15.55998452 +0000 UTC m=+3.960957751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:14.560060 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.560029    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:15.560018953 +0000 UTC m=+3.960992178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:14.765320 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.765274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:14.765475 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.765454    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:14.765537 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.765485    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:14.765537 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.765503    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:14.765607 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:14.765567    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:15.765551662 +0000 UTC m=+4.166524887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:14.982402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.982294    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:11:14 +0000 UTC" deadline="2028-03-09 17:39:59.572634906 +0000 UTC"
Sep 15 21:16:14.982402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:14.982352    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12980h23m44.590286865s"
Sep 15 21:16:15.069125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.069092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:15.069314 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.069214    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:15.201061 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.201028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c054e39_ac62_4c9e_8ed6_8fcc550f4d58.slice/crio-3240a3888f3149e4159adf8ada33b58875fa29bedd4884312342c3182d845e51 WatchSource:0}: Error finding container 3240a3888f3149e4159adf8ada33b58875fa29bedd4884312342c3182d845e51: Status 404 returned error can't find the container with id 3240a3888f3149e4159adf8ada33b58875fa29bedd4884312342c3182d845e51
Sep 15 21:16:15.202669 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.202562    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5e4f3e_9cbb_40d5_8cef_a3252987d135.slice/crio-16d1656c69279bbf87b4f887588203df916bd5403375d281d11781571bc54ace WatchSource:0}: Error finding container 16d1656c69279bbf87b4f887588203df916bd5403375d281d11781571bc54ace: Status 404 returned error can't find the container with id 16d1656c69279bbf87b4f887588203df916bd5403375d281d11781571bc54ace
Sep 15 21:16:15.204165 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.204091    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c74e18b_055c_4cbe_87a4_66683c7dbc19.slice/crio-59759987c041d3e5a9cecd7d6d061b94c7264f20cb5881b953f4d96e4a56e8ab WatchSource:0}: Error finding container 59759987c041d3e5a9cecd7d6d061b94c7264f20cb5881b953f4d96e4a56e8ab: Status 404 returned error can't find the container with id 59759987c041d3e5a9cecd7d6d061b94c7264f20cb5881b953f4d96e4a56e8ab
Sep 15 21:16:15.207502 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.207476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c05d54_7e85_40e1_8b84_94efb601d3e3.slice/crio-bac975e13bbd93579b98e44f06a3b55e8598d064acadc6a28d5e9e0dd2766610 WatchSource:0}: Error finding container bac975e13bbd93579b98e44f06a3b55e8598d064acadc6a28d5e9e0dd2766610: Status 404 returned error can't find the container with id bac975e13bbd93579b98e44f06a3b55e8598d064acadc6a28d5e9e0dd2766610
Sep 15 21:16:15.208952 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.208927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46f5857_c346_4595_a2a4_68ed885c4ce3.slice/crio-89b3ec1819d7691e03b0380a4c816216b7eb06d31318ae925383749fbf85acc6 WatchSource:0}: Error finding container 89b3ec1819d7691e03b0380a4c816216b7eb06d31318ae925383749fbf85acc6: Status 404 returned error can't find the container with id 89b3ec1819d7691e03b0380a4c816216b7eb06d31318ae925383749fbf85acc6
Sep 15 21:16:15.210926 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.210900    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d421be5_c692_4b4e_9510_ad6741f05228.slice/crio-bde47e543b2a8f664941d1988416e9a56de272fcbf04d13730c7da0d2431a84e WatchSource:0}: Error finding container bde47e543b2a8f664941d1988416e9a56de272fcbf04d13730c7da0d2431a84e: Status 404 returned error can't find the container with id bde47e543b2a8f664941d1988416e9a56de272fcbf04d13730c7da0d2431a84e
Sep 15 21:16:15.213104 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.212392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34463d15_8b3b_489d_b214_ba7573a8f230.slice/crio-cbb1633ab8119cd888ebd3a7950d7bdaa8bba9d738af8fb1e4e63bb0b53777f6 WatchSource:0}: Error finding container cbb1633ab8119cd888ebd3a7950d7bdaa8bba9d738af8fb1e4e63bb0b53777f6: Status 404 returned error can't find the container with id cbb1633ab8119cd888ebd3a7950d7bdaa8bba9d738af8fb1e4e63bb0b53777f6
Sep 15 21:16:15.213538 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.213505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb312a59f_32c2_4483_8147_28b5e2f0df15.slice/crio-8de092495356ba5326eca51cc98fdc5a33e5c636b45cdb6ac1e2777e1a47ec32 WatchSource:0}: Error finding container 8de092495356ba5326eca51cc98fdc5a33e5c636b45cdb6ac1e2777e1a47ec32: Status 404 returned error can't find the container with id 8de092495356ba5326eca51cc98fdc5a33e5c636b45cdb6ac1e2777e1a47ec32
Sep 15 21:16:15.214084 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:16:15.213973    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31cf26a_9cfe_4fe8_9aab_dfba02d0b1d2.slice/crio-390266b14f548b4523ddef8f69f166d8e57499def9bc8b1ec4df7682e9d57bc7 WatchSource:0}: Error finding container 390266b14f548b4523ddef8f69f166d8e57499def9bc8b1ec4df7682e9d57bc7: Status 404 returned error can't find the container with id 390266b14f548b4523ddef8f69f166d8e57499def9bc8b1ec4df7682e9d57bc7
Sep 15 21:16:15.351247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.351217    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:16:15.571818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.571785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:15.571989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.571830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:15.571989 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.571934    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:15.571989 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.571948    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:15.571989 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.571991    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.571978414 +0000 UTC m=+5.972951638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:15.572169 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.572016    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.571997584 +0000 UTC m=+5.972970808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:15.773634 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.773551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:15.773881 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.773764    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:15.773881 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.773789    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:15.773881 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.773803    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:15.773881 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:15.773861    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:17.773842297 +0000 UTC m=+6.174815526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:15.983169 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.983122    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:11:14 +0000 UTC" deadline="2028-02-23 08:01:35.906295102 +0000 UTC"
Sep 15 21:16:15.983169 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:15.983166    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12610h45m19.923133949s"
Sep 15 21:16:16.071549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.070838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:16.071549 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:16.070978    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:16.071549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.071350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:16.071549 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:16.071465    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:16.095378 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.095341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkkcg" event={"ID":"d46f5857-c346-4595-a2a4-68ed885c4ce3","Type":"ContainerStarted","Data":"89b3ec1819d7691e03b0380a4c816216b7eb06d31318ae925383749fbf85acc6"}
Sep 15 21:16:16.108501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.108464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h468b" event={"ID":"5c74e18b-055c-4cbe-87a4-66683c7dbc19","Type":"ContainerStarted","Data":"59759987c041d3e5a9cecd7d6d061b94c7264f20cb5881b953f4d96e4a56e8ab"}
Sep 15 21:16:16.111261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.111229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xwltq" event={"ID":"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2","Type":"ContainerStarted","Data":"390266b14f548b4523ddef8f69f166d8e57499def9bc8b1ec4df7682e9d57bc7"}
Sep 15 21:16:16.122754 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.122671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerStarted","Data":"cbb1633ab8119cd888ebd3a7950d7bdaa8bba9d738af8fb1e4e63bb0b53777f6"}
Sep 15 21:16:16.128969 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.128895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v4fxj" event={"ID":"1d421be5-c692-4b4e-9510-ad6741f05228","Type":"ContainerStarted","Data":"bde47e543b2a8f664941d1988416e9a56de272fcbf04d13730c7da0d2431a84e"}
Sep 15 21:16:16.139068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.138906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"bac975e13bbd93579b98e44f06a3b55e8598d064acadc6a28d5e9e0dd2766610"}
Sep 15 21:16:16.152531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.152442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vctmx" event={"ID":"cb5e4f3e-9cbb-40d5-8cef-a3252987d135","Type":"ContainerStarted","Data":"16d1656c69279bbf87b4f887588203df916bd5403375d281d11781571bc54ace"}
Sep 15 21:16:16.170350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.170316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf" event={"ID":"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58","Type":"ContainerStarted","Data":"3240a3888f3149e4159adf8ada33b58875fa29bedd4884312342c3182d845e51"}
Sep 15 21:16:16.183114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.183079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal" event={"ID":"54b69627995a64131f3948c3f8331567","Type":"ContainerStarted","Data":"1ee310e5163ba0f012eb16a1c2d947b2bdcc44bc6d85f64da341e4cd10eb8ab3"}
Sep 15 21:16:16.188460 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.187975    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b90c546182f4af06bbc93616788e0ad" containerID="b7c7f6644d6bf305f1869928742a225f1db8d35ecf860a5019b90b2dbfb20dd9" exitCode=0
Sep 15 21:16:16.188460 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.188130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal" event={"ID":"7b90c546182f4af06bbc93616788e0ad","Type":"ContainerDied","Data":"b7c7f6644d6bf305f1869928742a225f1db8d35ecf860a5019b90b2dbfb20dd9"}
Sep 15 21:16:16.198163 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.197900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-105.ec2.internal" podStartSLOduration=3.197880874 podStartE2EDuration="3.197880874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:16.196903272 +0000 UTC m=+4.597876520" watchObservedRunningTime="2026-09-15 21:16:16.197880874 +0000 UTC m=+4.598854123"
Sep 15 21:16:16.198584 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:16.198373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wltw9" event={"ID":"b312a59f-32c2-4483-8147-28b5e2f0df15","Type":"ContainerStarted","Data":"8de092495356ba5326eca51cc98fdc5a33e5c636b45cdb6ac1e2777e1a47ec32"}
Sep 15 21:16:17.069268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.069232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:17.069849 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.069382    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:17.214976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.214890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal" event={"ID":"7b90c546182f4af06bbc93616788e0ad","Type":"ContainerStarted","Data":"0afc88384609da65fee1686a9a46fd9d1020b4bfd9d774a823357d509c9f6726"}
Sep 15 21:16:17.231820 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.231760    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-105.ec2.internal" podStartSLOduration=4.231738996 podStartE2EDuration="4.231738996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:17.231657125 +0000 UTC m=+5.632630399" watchObservedRunningTime="2026-09-15 21:16:17.231738996 +0000 UTC m=+5.632712241"
Sep 15 21:16:17.590827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.590707    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:17.590827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.590779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:17.591046 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.590910    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:17.591046 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.590983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:21.590956423 +0000 UTC m=+9.991929678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:17.591157 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.591061    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:17.591157 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.591096    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:21.591084197 +0000 UTC m=+9.992057426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:17.792449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:17.792410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:17.792650 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.792629    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:17.792650 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.792649    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:17.792843 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.792662    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:17.792843 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:17.792742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:21.792702539 +0000 UTC m=+10.193675768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:18.071413 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:18.069493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:18.071413 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:18.069622    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:18.071413 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:18.070108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:18.071413 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:18.070216    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:19.069411 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:19.068934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:19.069411 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:19.069069    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:20.070107 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:20.070076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:20.070591 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:20.070192    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:20.070591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:20.070243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:20.070591 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:20.070370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:21.069927 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:21.069738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:21.069927 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.069884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:21.626551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:21.626510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:21.627023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:21.626573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:21.627023 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.626697    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:21.627023 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.626756    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:29.626744649 +0000 UTC m=+18.027717873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:21.627199 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.627071    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:21.627199 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.627108    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:29.627097427 +0000 UTC m=+18.028070650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:21.828897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:21.828842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:21.829171 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.829059    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:21.829171 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.829084    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:21.829171 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.829096    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:21.829171 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:21.829157    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:29.829138711 +0000 UTC m=+18.230111954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:22.070814 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:22.070137    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:22.070814 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:22.070245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:22.070814 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:22.070644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:22.070814 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:22.070763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:23.069323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:23.069285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:23.069816 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:23.069446    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:24.069487 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:24.069453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:24.069989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:24.069466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:24.069989 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:24.069587    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:24.069989 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:24.069760    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:25.069658 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:25.069617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:25.070146 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:25.069783    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:26.069503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:26.069408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:26.069664 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:26.069408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:26.069664 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:26.069559    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:26.069664 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:26.069637    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:27.069115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:27.069075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:27.069271 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:27.069216    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:28.069767 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:28.069728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:28.069767 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:28.069767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:28.070243 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:28.069853    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:28.070243 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:28.069985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:29.069167 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:29.069136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:29.069407 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.069239    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:29.688614 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:29.688575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:29.688614 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:29.688624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:29.689128 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.688766    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:29.689128 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.688844    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.688823602 +0000 UTC m=+34.089796854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:29.689128 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.688776    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:29.689128 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.688913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.688900493 +0000 UTC m=+34.089873729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:29.891170 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:29.891128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:29.891412 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.891314    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:29.891412 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.891338    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:29.891412 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.891349    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:29.891412 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:29.891401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:16:45.891386798 +0000 UTC m=+34.292360023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:30.069562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:30.069444    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:30.069746 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:30.069681    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:30.072275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:30.069960    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:30.072275 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:30.070167    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:31.069771 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:31.069735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:31.070249 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:31.069861    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:32.070154 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:32.070124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:32.070574 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:32.070242    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:32.070574 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:32.070294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:32.070574 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:32.070416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:33.069586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:33.069550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:33.069740 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:33.069662    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:34.069591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.069366    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:34.070355 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.069392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:34.070355 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:34.069735    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:34.070355 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:34.069822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:34.260163 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.260084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wltw9" event={"ID":"b312a59f-32c2-4483-8147-28b5e2f0df15","Type":"ContainerStarted","Data":"f26a45d6e9cf8df752b3d344533c03f5d8f612821d80a3022c5d5b6c3404433b"}
Sep 15 21:16:34.261847 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.261819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkkcg" event={"ID":"d46f5857-c346-4595-a2a4-68ed885c4ce3","Type":"ContainerStarted","Data":"19d6c48f009e6822fd859cdb482d5fe2f34774b416a47a9ef5f1dbd9378788eb"}
Sep 15 21:16:34.263509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.263485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h468b" event={"ID":"5c74e18b-055c-4cbe-87a4-66683c7dbc19","Type":"ContainerStarted","Data":"a794d491d6f8cde670193772ebdf579828dedad56c880bf5461723892bbdaa2f"}
Sep 15 21:16:34.265258 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.265224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xwltq" event={"ID":"a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2","Type":"ContainerStarted","Data":"496573d413a9268fdfbad2a133b627a5b991e835e2cb3fcebcd63ee75d15e2d0"}
Sep 15 21:16:34.266871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.266850    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="23dd96fab1f8def75c14e10beb4bfa446fd1715a3a01cd473bebadd69a4162b0" exitCode=0
Sep 15 21:16:34.266967 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.266918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"23dd96fab1f8def75c14e10beb4bfa446fd1715a3a01cd473bebadd69a4162b0"}
Sep 15 21:16:34.268566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.268540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v4fxj" event={"ID":"1d421be5-c692-4b4e-9510-ad6741f05228","Type":"ContainerStarted","Data":"c3f6deafd5945ba6a91f366b4a6514a1e1992d7aa6144494ba16202229b48135"}
Sep 15 21:16:34.271509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"9db7f0c4389a44deca23a846797b6c18c7564975fbf1f5be33a05cc6302f66b7"}
Sep 15 21:16:34.271610 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"d544c5403d7259981c77262f03cbfb1e283ff1c8d1ba22bfac6d9555c6f8ac52"}
Sep 15 21:16:34.271610 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"6eea928547531d59e98bdb36d9f28819f0580f72d9b24baa71e44ab68f55191b"}
Sep 15 21:16:34.271691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"d62ebfbf1bc0bdd1e204ccdb5353bff9512e6f46a64d99bca61a376e1dc5f5ca"}
Sep 15 21:16:34.271691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"31c6b7f4a3ff09fcdf41a3416190a52496d2016cf4424107cab819f9ed2563f1"}
Sep 15 21:16:34.271691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.271687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"55b0e7855586a4a23c99f7cfa7a475b9731a1ef56894cd7a9da09f783b22dcf9"}
Sep 15 21:16:34.272914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.272886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf" event={"ID":"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58","Type":"ContainerStarted","Data":"d921ae6c2cd3b06be4476f4dff3114ecdddf45726241574d67b0b7a750278182"}
Sep 15 21:16:34.295803 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.295758    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wltw9" podStartSLOduration=11.716790885 podStartE2EDuration="22.295745046s" totalImagesPullingTime="10.578954161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.276681485 +0000 UTC m=+22.677654731" watchObservedRunningTime="2026-09-15 21:16:34.295745046 +0000 UTC m=+22.696718289"
Sep 15 21:16:34.310270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.310233    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h468b" podStartSLOduration=4.249913251 podStartE2EDuration="22.310221495s" totalImagesPullingTime="18.060308244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.310022987 +0000 UTC m=+22.710996243" watchObservedRunningTime="2026-09-15 21:16:34.310221495 +0000 UTC m=+22.711194741"
Sep 15 21:16:34.326528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.326488    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xkkcg" podStartSLOduration=4.32494629 podStartE2EDuration="22.326476818s" totalImagesPullingTime="18.001530528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.326240201 +0000 UTC m=+22.727213448" watchObservedRunningTime="2026-09-15 21:16:34.326476818 +0000 UTC m=+22.727450064"
Sep 15 21:16:34.354640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.354594    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v4fxj" podStartSLOduration=4.357349707 podStartE2EDuration="22.354581143s" totalImagesPullingTime="17.997231436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.354573088 +0000 UTC m=+22.755546334" watchObservedRunningTime="2026-09-15 21:16:34.354581143 +0000 UTC m=+22.755554418"
Sep 15 21:16:34.354798 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.354779    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xwltq" podStartSLOduration=4.360679689 podStartE2EDuration="22.354773511s" totalImagesPullingTime="17.994093822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:34.342583369 +0000 UTC m=+22.743556614" watchObservedRunningTime="2026-09-15 21:16:34.354773511 +0000 UTC m=+22.755746756"
Sep 15 21:16:34.899128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.898969    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:16:34.996333 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.996226    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:16:34.899122077Z","UUID":"90d65b38-9929-409b-a69d-31f80ba23d11","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:16:34.997845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.997824    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:16:34.997845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:34.997856    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:16:35.069174 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:35.069145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:35.069321 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:35.069245    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:35.277150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:35.277060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vctmx" event={"ID":"cb5e4f3e-9cbb-40d5-8cef-a3252987d135","Type":"ContainerStarted","Data":"bf32b490d1d479f79e14fcb52a95e76ae45f5ad8f9fd7bf1de98675fb61c9570"}
Sep 15 21:16:35.279371 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:35.279343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf" event={"ID":"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58","Type":"ContainerStarted","Data":"2c09188b9956b720dae50107b7419d92bbbed5354606bd1287fe8dd318d7a608"}
Sep 15 21:16:35.293457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:35.293403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vctmx" podStartSLOduration=5.287743761 podStartE2EDuration="23.293382732s" totalImagesPullingTime="18.005638971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:35.293287428 +0000 UTC m=+23.694260676" watchObservedRunningTime="2026-09-15 21:16:35.293382732 +0000 UTC m=+23.694355980"
Sep 15 21:16:36.073095 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.073064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:36.073274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.073173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:36.073274 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:36.073185    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:36.073390 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:36.073268    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:36.086704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.086678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:36.087185 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.087170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:36.286549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.286509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"851a2bec638272ae2aa4527008e8218eb7782a0cba7e4e56e859142c1276e1b2"}
Sep 15 21:16:36.287052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.286854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:36.287397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:36.287379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wltw9"
Sep 15 21:16:37.069675 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:37.069644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:37.069857 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:37.069789    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:38.070072 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:38.070035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:38.070489 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:38.070182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:38.070489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:38.070233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:38.070489 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:38.070347    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:38.293480 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:38.293362    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf" event={"ID":"2c054e39-ac62-4c9e-8ed6-8fcc550f4d58","Type":"ContainerStarted","Data":"bfe061acb4cc54ae6abdd0580b2f120af8b319b52609bcd279eda5aeda59b066"}
Sep 15 21:16:38.311093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:38.311034    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gtxqf" podStartSLOduration=4.618844399 podStartE2EDuration="26.311013995s" totalImagesPullingTime="21.692169596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:38.310935 +0000 UTC m=+26.711908247" watchObservedRunningTime="2026-09-15 21:16:38.311013995 +0000 UTC m=+26.711987240"
Sep 15 21:16:39.069460 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:39.069425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:39.069623 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:39.069551    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:40.069903 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.069708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:40.070585 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.069797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:40.070585 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:40.069997    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:40.070585 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:40.070087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:40.298863 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.298834    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="d15473860be324f9e96628816c6b6be1bbdbccf5d91ec502c666dfb1759c3ec5" exitCode=0
Sep 15 21:16:40.299034 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.298921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"d15473860be324f9e96628816c6b6be1bbdbccf5d91ec502c666dfb1759c3ec5"}
Sep 15 21:16:40.302925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.302902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" event={"ID":"51c05d54-7e85-40e1-8b84-94efb601d3e3","Type":"ContainerStarted","Data":"601b79e5dd1e3a02162b75a148a6d7be3a13c33c5acb0135800c5d8edbad0a02"}
Sep 15 21:16:40.303159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.303138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:40.317843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.317823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:40.346920 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:40.346828    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g" podStartSLOduration=10.294474734 podStartE2EDuration="28.346813915s" totalImagesPullingTime="18.052339181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:40.346173164 +0000 UTC m=+28.747146410" watchObservedRunningTime="2026-09-15 21:16:40.346813915 +0000 UTC m=+28.747787160"
Sep 15 21:16:41.069910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.069699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:41.070539 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:41.070018    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:41.082653 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.082627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:41.265380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.265349    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z4tbr"]
Sep 15 21:16:41.271195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.271167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd7zt"]
Sep 15 21:16:41.271339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.271264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:41.271398 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:41.271353    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:41.271850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.271827    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krh4j"]
Sep 15 21:16:41.271973 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.271925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:41.272059 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:41.272032    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:41.304906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.304841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:41.305059 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:41.304949    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:41.305350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.305329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:41.320465 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:41.320386    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:16:42.310221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:42.310188    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="7376726b02f4934906e4b8f4a3f5eb2eacf47eec4129c33e8504a206ce861183" exitCode=0
Sep 15 21:16:42.310648 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:42.310274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"7376726b02f4934906e4b8f4a3f5eb2eacf47eec4129c33e8504a206ce861183"}
Sep 15 21:16:43.069894 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:43.069862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:43.070066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:43.069862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:43.070066 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:43.069972    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:43.070066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:43.069862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:43.070066 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:43.070040    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:43.070209 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:43.070116    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:45.069779 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.069585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:45.070221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.069585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:45.070221 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.069858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd7zt" podUID="5d328c0f-2630-4508-928d-a1f642735acf"
Sep 15 21:16:45.070221 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.069971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krh4j" podUID="9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd"
Sep 15 21:16:45.070221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.069592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:45.070221 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.070069    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z4tbr" podUID="7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94"
Sep 15 21:16:45.320708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.320623    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="f9c423db71460079658960001623d59ed1c08e330336442d4e2d08ae3b9bbbe8" exitCode=0
Sep 15 21:16:45.320708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.320685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"f9c423db71460079658960001623d59ed1c08e330336442d4e2d08ae3b9bbbe8"}
Sep 15 21:16:45.708219 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.708190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:45.708374 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.708226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:45.708374 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.708309    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:45.708374 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.708319    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:45.708374 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.708357    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret podName:7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:17.708344441 +0000 UTC m=+66.109317666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret") pod "global-pull-secret-syncer-z4tbr" (UID: "7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:16:45.708374 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.708370    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs podName:9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd nodeName:}" failed. No retries permitted until 2026-09-15 21:17:17.708364199 +0000 UTC m=+66.109337423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs") pod "network-metrics-daemon-krh4j" (UID: "9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:16:45.803038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.803015    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-105.ec2.internal" event="NodeReady"
Sep 15 21:16:45.803175 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.803146    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:16:45.909825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:45.909794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:45.910019 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.909931    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:16:45.910019 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.909952    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:16:45.910019 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.909963    2578 projected.go:196] Error preparing data for projected volume kube-api-access-xbvd6 for pod openshift-network-diagnostics/network-check-target-pd7zt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:45.910130 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:16:45.910023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6 podName:5d328c0f-2630-4508-928d-a1f642735acf nodeName:}" failed. No retries permitted until 2026-09-15 21:17:17.910006326 +0000 UTC m=+66.310979564 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbvd6" (UniqueName: "kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6") pod "network-check-target-pd7zt" (UID: "5d328c0f-2630-4508-928d-a1f642735acf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:16:46.732276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:46.732247    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xwltq_a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2/dns-node-resolver/0.log"
Sep 15 21:16:47.069569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.069483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:16:47.069740 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.069483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:16:47.069861 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.069486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:16:47.075914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.075882    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:16:47.075914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.075908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\""
Sep 15 21:16:47.076089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.075938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:16:47.079178 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.079150    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:16:47.079323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.079303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\""
Sep 15 21:16:47.079992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.079975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:16:47.689011 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:47.688988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v4fxj_1d421be5-c692-4b4e-9510-ad6741f05228/node-ca/0.log"
Sep 15 21:16:50.334989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:50.334956    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="eb6c7898489c8d6b54753fb14508bd31eeca98dafa436f7c3c977ce7ab47d076" exitCode=0
Sep 15 21:16:50.335488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:50.335032    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"eb6c7898489c8d6b54753fb14508bd31eeca98dafa436f7c3c977ce7ab47d076"}
Sep 15 21:16:51.342165 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:51.342131    2578 generic.go:356] "Generic (PLEG): container finished" podID="34463d15-8b3b-489d-b214-ba7573a8f230" containerID="cf20a4f501339324b3b93daae3ccbca2559b6fabf10a9035568e9ba2a9879cd9" exitCode=0
Sep 15 21:16:51.342676 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:51.342188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerDied","Data":"cf20a4f501339324b3b93daae3ccbca2559b6fabf10a9035568e9ba2a9879cd9"}
Sep 15 21:16:52.347622 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:52.347583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" event={"ID":"34463d15-8b3b-489d-b214-ba7573a8f230","Type":"ContainerStarted","Data":"dd2798334e14cc769fe0d0af5d2a6178711dac9db308a8ad8a74bf1c25742fd0"}
Sep 15 21:16:52.375481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:16:52.375420    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9ngcv" podStartSLOduration=9.159061108 podStartE2EDuration="40.37540546s" totalImagesPullingTime="31.216344352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:16:52.374923347 +0000 UTC m=+40.775896593" watchObservedRunningTime="2026-09-15 21:16:52.37540546 +0000 UTC m=+40.776378705"
Sep 15 21:17:08.229586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.229553    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lgq7k"]
Sep 15 21:17:08.232155 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.232125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.234760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.234734    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k7wrd\""
Sep 15 21:17:08.239459 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.239439    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6dw79"]
Sep 15 21:17:08.241430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.241415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.243509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.243489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:17:08.243584 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.243528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ztstg\""
Sep 15 21:17:08.243584 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.243497    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:17:08.244129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.244114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:17:08.244129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.244126    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:17:08.252317 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.252300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:17:08.257481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.257463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgq7k"]
Sep 15 21:17:08.263976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.263956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dw79"]
Sep 15 21:17:08.265447 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce900c-bd00-4547-9b78-f0b5635bd252-tmp-dir\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.265551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cce900c-bd00-4547-9b78-f0b5635bd252-metrics-tls\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.265551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/4cce900c-bd00-4547-9b78-f0b5635bd252-kube-api-access-crwbc\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.265551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-cert\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.265551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cce900c-bd00-4547-9b78-f0b5635bd252-config-volume\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.265705 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.265576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6p2p\" (UniqueName: \"kubernetes.io/projected/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-kube-api-access-l6p2p\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.297611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.297586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ddc8578db-7l7rv"]
Sep 15 21:17:08.299486 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.299471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.301834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.301815    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 21:17:08.304305 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.304284    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 21:17:08.305447 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.305186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7rm"]
Sep 15 21:17:08.305778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.305760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 21:17:08.306150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.306133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fzsgv\""
Sep 15 21:17:08.307108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.307089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.312683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.312663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:17:08.313195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.313179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p6jkf\""
Sep 15 21:17:08.313269 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.313186    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:17:08.313319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.313282    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:17:08.313479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.313466    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:17:08.315364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.315349    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 21:17:08.328022 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.328001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ddc8578db-7l7rv"]
Sep 15 21:17:08.342356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.342328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7rm"]
Sep 15 21:17:08.366203 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-image-registry-private-configuration\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366203 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmprf\" (UniqueName: \"kubernetes.io/projected/49f3aed7-634d-4cac-885e-0242bf955c33-kube-api-access-mmprf\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-cert\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cce900c-bd00-4547-9b78-f0b5635bd252-config-volume\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-certificates\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6p2p\" (UniqueName: \"kubernetes.io/projected/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-kube-api-access-l6p2p\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fc94d5-a8f9-459f-8480-d19dcf8b879f-ca-trust-extracted\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-installation-pull-secrets\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49f3aed7-634d-4cac-885e-0242bf955c33-data-volume\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce900c-bd00-4547-9b78-f0b5635bd252-tmp-dir\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvdz\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-kube-api-access-nlvdz\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-bound-sa-token\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49f3aed7-634d-4cac-885e-0242bf955c33-crio-socket\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cce900c-bd00-4547-9b78-f0b5635bd252-metrics-tls\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-tls\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-trusted-ca\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49f3aed7-634d-4cac-885e-0242bf955c33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.366750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49f3aed7-634d-4cac-885e-0242bf955c33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.367119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/4cce900c-bd00-4547-9b78-f0b5635bd252-kube-api-access-crwbc\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.367119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cce900c-bd00-4547-9b78-f0b5635bd252-tmp-dir\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.367119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.366815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cce900c-bd00-4547-9b78-f0b5635bd252-config-volume\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.370191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.370171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cce900c-bd00-4547-9b78-f0b5635bd252-metrics-tls\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.370289 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.370265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-cert\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.390090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.390066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwbc\" (UniqueName: \"kubernetes.io/projected/4cce900c-bd00-4547-9b78-f0b5635bd252-kube-api-access-crwbc\") pod \"dns-default-6dw79\" (UID: \"4cce900c-bd00-4547-9b78-f0b5635bd252\") " pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.391590 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.391574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6p2p\" (UniqueName: \"kubernetes.io/projected/4b2750c1-af2a-4107-bdbb-efe9eb5e8763-kube-api-access-l6p2p\") pod \"ingress-canary-lgq7k\" (UID: \"4b2750c1-af2a-4107-bdbb-efe9eb5e8763\") " pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.467440 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvdz\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-kube-api-access-nlvdz\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.467594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-bound-sa-token\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.467594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49f3aed7-634d-4cac-885e-0242bf955c33-crio-socket\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.467594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49f3aed7-634d-4cac-885e-0242bf955c33-crio-socket\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.467594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-tls\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.467594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-trusted-ca\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49f3aed7-634d-4cac-885e-0242bf955c33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49f3aed7-634d-4cac-885e-0242bf955c33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-image-registry-private-configuration\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmprf\" (UniqueName: \"kubernetes.io/projected/49f3aed7-634d-4cac-885e-0242bf955c33-kube-api-access-mmprf\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-certificates\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fc94d5-a8f9-459f-8480-d19dcf8b879f-ca-trust-extracted\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-installation-pull-secrets\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.467990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49f3aed7-634d-4cac-885e-0242bf955c33-data-volume\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468426 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.468245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00fc94d5-a8f9-459f-8480-d19dcf8b879f-ca-trust-extracted\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468426 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.468299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49f3aed7-634d-4cac-885e-0242bf955c33-data-volume\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468426 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.468328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49f3aed7-634d-4cac-885e-0242bf955c33-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.468635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.468611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-certificates\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.468827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.468806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00fc94d5-a8f9-459f-8480-d19dcf8b879f-trusted-ca\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.470145 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.470123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-image-registry-private-configuration\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.470246 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.470201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-registry-tls\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.470845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.470816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49f3aed7-634d-4cac-885e-0242bf955c33-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.470934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.470816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00fc94d5-a8f9-459f-8480-d19dcf8b879f-installation-pull-secrets\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.482524 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.482472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvdz\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-kube-api-access-nlvdz\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.483012 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.482996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00fc94d5-a8f9-459f-8480-d19dcf8b879f-bound-sa-token\") pod \"image-registry-5ddc8578db-7l7rv\" (UID: \"00fc94d5-a8f9-459f-8480-d19dcf8b879f\") " pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.486965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.486944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmprf\" (UniqueName: \"kubernetes.io/projected/49f3aed7-634d-4cac-885e-0242bf955c33-kube-api-access-mmprf\") pod \"insights-runtime-extractor-mb7rm\" (UID: \"49f3aed7-634d-4cac-885e-0242bf955c33\") " pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.542008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.541978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lgq7k"
Sep 15 21:17:08.548707 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.548682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:08.608491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.608405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:08.615667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.615633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7rm"
Sep 15 21:17:08.830172 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.830097    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lgq7k"]
Sep 15 21:17:08.833698 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:08.833668    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2750c1_af2a_4107_bdbb_efe9eb5e8763.slice/crio-6a73df6f0b3c4e0a40c9b4c94524e677d9b9bf8759b6fb835b497d189b3c5bb8 WatchSource:0}: Error finding container 6a73df6f0b3c4e0a40c9b4c94524e677d9b9bf8759b6fb835b497d189b3c5bb8: Status 404 returned error can't find the container with id 6a73df6f0b3c4e0a40c9b4c94524e677d9b9bf8759b6fb835b497d189b3c5bb8
Sep 15 21:17:08.834283 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.834262    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dw79"]
Sep 15 21:17:08.839164 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:08.839141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cce900c_bd00_4547_9b78_f0b5635bd252.slice/crio-9a7dadb188db0ede96da34d66a1925f0bf0215a9c3356227b4313b1584662091 WatchSource:0}: Error finding container 9a7dadb188db0ede96da34d66a1925f0bf0215a9c3356227b4313b1584662091: Status 404 returned error can't find the container with id 9a7dadb188db0ede96da34d66a1925f0bf0215a9c3356227b4313b1584662091
Sep 15 21:17:08.893479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.893449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7rm"]
Sep 15 21:17:08.896737 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:08.896688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ddc8578db-7l7rv"]
Sep 15 21:17:08.896892 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:08.896870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f3aed7_634d_4cac_885e_0242bf955c33.slice/crio-e4e7f918eb5e29026b9086cb08634b7869a01befcd9929309e039a7b9adf4aaa WatchSource:0}: Error finding container e4e7f918eb5e29026b9086cb08634b7869a01befcd9929309e039a7b9adf4aaa: Status 404 returned error can't find the container with id e4e7f918eb5e29026b9086cb08634b7869a01befcd9929309e039a7b9adf4aaa
Sep 15 21:17:08.899851 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:08.899832    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fc94d5_a8f9_459f_8480_d19dcf8b879f.slice/crio-b06f58debc94fc9f19dc415e16a9a14f9851511c418e12604066ae52edeb685b WatchSource:0}: Error finding container b06f58debc94fc9f19dc415e16a9a14f9851511c418e12604066ae52edeb685b: Status 404 returned error can't find the container with id b06f58debc94fc9f19dc415e16a9a14f9851511c418e12604066ae52edeb685b
Sep 15 21:17:09.388304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.388159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7rm" event={"ID":"49f3aed7-634d-4cac-885e-0242bf955c33","Type":"ContainerStarted","Data":"aa26f5975db28b79655c309c53917de68efbe9ced86fb077a36d7b2326bbec51"}
Sep 15 21:17:09.388304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.388235    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7rm" event={"ID":"49f3aed7-634d-4cac-885e-0242bf955c33","Type":"ContainerStarted","Data":"e4e7f918eb5e29026b9086cb08634b7869a01befcd9929309e039a7b9adf4aaa"}
Sep 15 21:17:09.391472 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.391329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv" event={"ID":"00fc94d5-a8f9-459f-8480-d19dcf8b879f","Type":"ContainerStarted","Data":"9317a892c109e4b46ae780b74f0f777280366f4a8f316b2030799457dd62ac58"}
Sep 15 21:17:09.391472 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.391386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv" event={"ID":"00fc94d5-a8f9-459f-8480-d19dcf8b879f","Type":"ContainerStarted","Data":"b06f58debc94fc9f19dc415e16a9a14f9851511c418e12604066ae52edeb685b"}
Sep 15 21:17:09.391472 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.391439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:09.393199 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.393145    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgq7k" event={"ID":"4b2750c1-af2a-4107-bdbb-efe9eb5e8763","Type":"ContainerStarted","Data":"6a73df6f0b3c4e0a40c9b4c94524e677d9b9bf8759b6fb835b497d189b3c5bb8"}
Sep 15 21:17:09.395302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.395243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw79" event={"ID":"4cce900c-bd00-4547-9b78-f0b5635bd252","Type":"ContainerStarted","Data":"9a7dadb188db0ede96da34d66a1925f0bf0215a9c3356227b4313b1584662091"}
Sep 15 21:17:09.417173 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:09.416973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv" podStartSLOduration=1.416953069 podStartE2EDuration="1.416953069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:09.415250354 +0000 UTC m=+57.816223602" watchObservedRunningTime="2026-09-15 21:17:09.416953069 +0000 UTC m=+57.817926313"
Sep 15 21:17:10.400962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:10.400871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7rm" event={"ID":"49f3aed7-634d-4cac-885e-0242bf955c33","Type":"ContainerStarted","Data":"a7157932c997345649f0651797c5db31dcf44ebfb49f57b7cca19367ad718481"}
Sep 15 21:17:13.325304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.325273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75t4g"
Sep 15 21:17:13.414566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.414532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7rm" event={"ID":"49f3aed7-634d-4cac-885e-0242bf955c33","Type":"ContainerStarted","Data":"257888eba38e8cd952c725965f2ece3cb30ba4207eeef799a65b4ccb24880da7"}
Sep 15 21:17:13.416113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.416089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lgq7k" event={"ID":"4b2750c1-af2a-4107-bdbb-efe9eb5e8763","Type":"ContainerStarted","Data":"a3f8a15ed3ef3bb40f94aed6619a56eee8403c393e19b61fba62bf717b1bc97c"}
Sep 15 21:17:13.417842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.417821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw79" event={"ID":"4cce900c-bd00-4547-9b78-f0b5635bd252","Type":"ContainerStarted","Data":"e4ce119eda078e40bb0ca4e9cfe53645074b4700aafca1f60161459600cbc22b"}
Sep 15 21:17:13.417946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.417861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw79" event={"ID":"4cce900c-bd00-4547-9b78-f0b5635bd252","Type":"ContainerStarted","Data":"da1e171a656c4ee1131bf1bf226c16d89008ac15854a8551acebb88a94966e83"}
Sep 15 21:17:13.417946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.417898    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:13.435909 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.435854    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mb7rm" podStartSLOduration=2.071738076 podStartE2EDuration="5.435835751s" totalImagesPullingTime="3.364097675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:17:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:13.435422208 +0000 UTC m=+61.836395455" watchObservedRunningTime="2026-09-15 21:17:13.435835751 +0000 UTC m=+61.836808997"
Sep 15 21:17:13.455843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.455774    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6dw79" podStartSLOduration=1.888764656 podStartE2EDuration="5.455755844s" totalImagesPullingTime="3.566991188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:13.454868768 +0000 UTC m=+61.855842015" watchObservedRunningTime="2026-09-15 21:17:13.455755844 +0000 UTC m=+61.856729084"
Sep 15 21:17:13.472076 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:13.472023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lgq7k" podStartSLOduration=1.8993525660000001 podStartE2EDuration="5.472006967s" totalImagesPullingTime="3.572654401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:13.471499541 +0000 UTC m=+61.872472788" watchObservedRunningTime="2026-09-15 21:17:13.472006967 +0000 UTC m=+61.872980212"
Sep 15 21:17:15.685975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.685944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"]
Sep 15 21:17:15.689542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.689525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:15.691278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.691259    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 21:17:15.691343 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.691331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rksnq\""
Sep 15 21:17:15.696967 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.696946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"]
Sep 15 21:17:15.722669 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.722644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pkdmd\" (UID: \"b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:15.823290 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.823256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pkdmd\" (UID: \"b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:15.825661 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.825630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pkdmd\" (UID: \"b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:15.997881 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:15.997836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:16.263837 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:16.263755    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"]
Sep 15 21:17:16.267263 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:16.267232    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f4ccfa_a9dd_4447_bc0d_6cb239cb6ebc.slice/crio-d82cc2f7a0cf353fa2edc4650b7d9ba72c8e5297e7e1799d3c6563e4d3c16c3b WatchSource:0}: Error finding container d82cc2f7a0cf353fa2edc4650b7d9ba72c8e5297e7e1799d3c6563e4d3c16c3b: Status 404 returned error can't find the container with id d82cc2f7a0cf353fa2edc4650b7d9ba72c8e5297e7e1799d3c6563e4d3c16c3b
Sep 15 21:17:16.431048 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:16.431011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd" event={"ID":"b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc","Type":"ContainerStarted","Data":"d82cc2f7a0cf353fa2edc4650b7d9ba72c8e5297e7e1799d3c6563e4d3c16c3b"}
Sep 15 21:17:17.736736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.736674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:17:17.737110 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.736789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:17:17.738893 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.738872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:17:17.738944 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.738875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:17:17.749424 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.749397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94-original-pull-secret\") pod \"global-pull-secret-syncer-z4tbr\" (UID: \"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94\") " pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:17:17.749611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.749401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd-metrics-certs\") pod \"network-metrics-daemon-krh4j\" (UID: \"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd\") " pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:17:17.938449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.938405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:17:17.940490 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.940472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:17:17.950543 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.950524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:17:17.961417 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.961392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvd6\" (UniqueName: \"kubernetes.io/projected/5d328c0f-2630-4508-928d-a1f642735acf-kube-api-access-xbvd6\") pod \"network-check-target-pd7zt\" (UID: \"5d328c0f-2630-4508-928d-a1f642735acf\") " pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:17:17.983731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.983705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssjn7\""
Sep 15 21:17:17.991685 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.991644    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7n8ch\""
Sep 15 21:17:17.992649 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.992634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krh4j"
Sep 15 21:17:17.995385 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:17.995367    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z4tbr"
Sep 15 21:17:18.000300 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.000282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:17:18.271043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.270952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z4tbr"]
Sep 15 21:17:18.274692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.274664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krh4j"]
Sep 15 21:17:18.274692 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:18.274676    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa081ed_af1f_44c1_b4ba_25ae8c2c2b94.slice/crio-8f9aef9acfd57a7aff8abb1c9f4174575fa8948c7bdb535a7d08b684fdd8e073 WatchSource:0}: Error finding container 8f9aef9acfd57a7aff8abb1c9f4174575fa8948c7bdb535a7d08b684fdd8e073: Status 404 returned error can't find the container with id 8f9aef9acfd57a7aff8abb1c9f4174575fa8948c7bdb535a7d08b684fdd8e073
Sep 15 21:17:18.275571 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.275549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd7zt"]
Sep 15 21:17:18.277652 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:18.277631    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8e4ca4_ae7f_4b7e_b319_844f937b3cdd.slice/crio-35ed161f52b4a566ecbc8671329221146360bfee9fea044f922d37609275fe9e WatchSource:0}: Error finding container 35ed161f52b4a566ecbc8671329221146360bfee9fea044f922d37609275fe9e: Status 404 returned error can't find the container with id 35ed161f52b4a566ecbc8671329221146360bfee9fea044f922d37609275fe9e
Sep 15 21:17:18.278472 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:18.278453    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d328c0f_2630_4508_928d_a1f642735acf.slice/crio-8a8bd16fe47046ac2e87b600a118f9a15c41145e449a03ca9af0b9ff5d28e8dd WatchSource:0}: Error finding container 8a8bd16fe47046ac2e87b600a118f9a15c41145e449a03ca9af0b9ff5d28e8dd: Status 404 returned error can't find the container with id 8a8bd16fe47046ac2e87b600a118f9a15c41145e449a03ca9af0b9ff5d28e8dd
Sep 15 21:17:18.439160 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.439123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd" event={"ID":"b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc","Type":"ContainerStarted","Data":"9620b6fe43e0ca6138dbfa8f01b184f30c803b6d3a69272826965bbd39685b59"}
Sep 15 21:17:18.439349 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.439281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:18.440525 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.440484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd7zt" event={"ID":"5d328c0f-2630-4508-928d-a1f642735acf","Type":"ContainerStarted","Data":"8a8bd16fe47046ac2e87b600a118f9a15c41145e449a03ca9af0b9ff5d28e8dd"}
Sep 15 21:17:18.441819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.441781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z4tbr" event={"ID":"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94","Type":"ContainerStarted","Data":"8f9aef9acfd57a7aff8abb1c9f4174575fa8948c7bdb535a7d08b684fdd8e073"}
Sep 15 21:17:18.443045 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.443019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krh4j" event={"ID":"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd","Type":"ContainerStarted","Data":"35ed161f52b4a566ecbc8671329221146360bfee9fea044f922d37609275fe9e"}
Sep 15 21:17:18.444655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.444638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd"
Sep 15 21:17:18.456880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.456835    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pkdmd" podStartSLOduration=2.327980071 podStartE2EDuration="3.456817983s" totalImagesPullingTime="1.128837912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:18.455937893 +0000 UTC m=+66.856911139" watchObservedRunningTime="2026-09-15 21:17:18.456817983 +0000 UTC m=+66.857791233"
Sep 15 21:17:18.748261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.748224    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"]
Sep 15 21:17:18.751936 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.751915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.753874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.753843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 21:17:18.753874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.753854    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:17:18.754120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.753854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 21:17:18.754120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.753849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:17:18.754341 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.754317    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:17:18.754452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.754433    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sq2rp\""
Sep 15 21:17:18.759803 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.759780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"]
Sep 15 21:17:18.843099 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.843062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.843275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.843109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4cx\" (UniqueName: \"kubernetes.io/projected/f96bf148-fca4-4bb4-99eb-278326cccd89-kube-api-access-cv4cx\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.843275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.843210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.843360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.843276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96bf148-fca4-4bb4-99eb-278326cccd89-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.943638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.943601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.943827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.943646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4cx\" (UniqueName: \"kubernetes.io/projected/f96bf148-fca4-4bb4-99eb-278326cccd89-kube-api-access-cv4cx\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.943827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.943685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.943827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.943734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96bf148-fca4-4bb4-99eb-278326cccd89-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.943998 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:17:18.943827    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 21:17:18.943998 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:17:18.943892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls podName:f96bf148-fca4-4bb4-99eb-278326cccd89 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:19.443875683 +0000 UTC m=+67.844848912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-rhs92" (UID: "f96bf148-fca4-4bb4-99eb-278326cccd89") : secret "prometheus-operator-tls" not found
Sep 15 21:17:18.944403 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.944379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96bf148-fca4-4bb4-99eb-278326cccd89-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.945974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.945954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:18.952088 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:18.952065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4cx\" (UniqueName: \"kubernetes.io/projected/f96bf148-fca4-4bb4-99eb-278326cccd89-kube-api-access-cv4cx\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:19.447412 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:19.447384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:19.449469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:19.449388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krh4j" event={"ID":"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd","Type":"ContainerStarted","Data":"0a26441ea16421a6072fccd7014b8b796c9470103fcaca3900406d7168607901"}
Sep 15 21:17:19.450190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:19.450078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f96bf148-fca4-4bb4-99eb-278326cccd89-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rhs92\" (UID: \"f96bf148-fca4-4bb4-99eb-278326cccd89\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:19.663860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:19.663760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"
Sep 15 21:17:19.938082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:19.938001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rhs92"]
Sep 15 21:17:19.941521 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:19.941478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bf148_fca4_4bb4_99eb_278326cccd89.slice/crio-358ca060902dbe5d891466073dd17601e27c1443934ad9703c2b6483f910cd7b WatchSource:0}: Error finding container 358ca060902dbe5d891466073dd17601e27c1443934ad9703c2b6483f910cd7b: Status 404 returned error can't find the container with id 358ca060902dbe5d891466073dd17601e27c1443934ad9703c2b6483f910cd7b
Sep 15 21:17:20.454643 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:20.454599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krh4j" event={"ID":"9f8e4ca4-ae7f-4b7e-b319-844f937b3cdd","Type":"ContainerStarted","Data":"b90e9230250eedfc0525c4e72033fdad940fd43c1030af30c2b264b61040a4d5"}
Sep 15 21:17:20.456057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:20.456024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92" event={"ID":"f96bf148-fca4-4bb4-99eb-278326cccd89","Type":"ContainerStarted","Data":"358ca060902dbe5d891466073dd17601e27c1443934ad9703c2b6483f910cd7b"}
Sep 15 21:17:20.471137 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:20.471074    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-krh4j" podStartSLOduration=67.509511639 podStartE2EDuration="1m8.471051391s" totalImagesPullingTime="961.539752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:20.469891014 +0000 UTC m=+68.870864252" watchObservedRunningTime="2026-09-15 21:17:20.471051391 +0000 UTC m=+68.872024658"
Sep 15 21:17:22.464165 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.464128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd7zt" event={"ID":"5d328c0f-2630-4508-928d-a1f642735acf","Type":"ContainerStarted","Data":"5d66c0f0f0a414b1da785a3baea316271f18699888e1ec83c431ad303c3e395f"}
Sep 15 21:17:22.464587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.464191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:17:22.465940 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.465915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92" event={"ID":"f96bf148-fca4-4bb4-99eb-278326cccd89","Type":"ContainerStarted","Data":"bc44c640050da6761e14cae8c719d0d326cf94354de0549bc6e06c58e615e47f"}
Sep 15 21:17:22.466056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.465963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92" event={"ID":"f96bf148-fca4-4bb4-99eb-278326cccd89","Type":"ContainerStarted","Data":"75ca5e0fdb288cfcc2c6f8f3ef572e0acf03f71434278dba863f9f4e84ca73c6"}
Sep 15 21:17:22.480672 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.480622    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pd7zt" podStartSLOduration=66.78996109 podStartE2EDuration="1m10.480607782s" totalImagesPullingTime="3.690646692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:22.479566893 +0000 UTC m=+70.880540140" watchObservedRunningTime="2026-09-15 21:17:22.480607782 +0000 UTC m=+70.881581029"
Sep 15 21:17:22.505353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:22.505285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rhs92" podStartSLOduration=2.481911409 podStartE2EDuration="4.505262624s" totalImagesPullingTime="2.023351215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:22.502608628 +0000 UTC m=+70.903581873" watchObservedRunningTime="2026-09-15 21:17:22.505262624 +0000 UTC m=+70.906235871"
Sep 15 21:17:23.423137 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:23.423097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6dw79"
Sep 15 21:17:24.110202 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.110168    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"]
Sep 15 21:17:24.145061 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.145024    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"]
Sep 15 21:17:24.145226 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.145078    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"]
Sep 15 21:17:24.145226 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.145173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.146662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.146637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 21:17:24.146837 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.146812    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h7rlw\""
Sep 15 21:17:24.146993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.146972    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:17:24.156680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.156655    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ws9f2"]
Sep 15 21:17:24.156837 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.156806    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.158962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.158940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:17:24.159405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.159228    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 21:17:24.159405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.159352    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 21:17:24.159405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.159351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j7ctd\""
Sep 15 21:17:24.179016 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.178984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"]
Sep 15 21:17:24.179160 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.179131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.180929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.180904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:17:24.181057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.180915    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hp94g\""
Sep 15 21:17:24.181057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.181042    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:17:24.182166 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.182129    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:17:24.183591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.183699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.183699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpk6\" (UniqueName: \"kubernetes.io/projected/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-api-access-7jpk6\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61c89006-c0e2-46db-a8ab-16f7a63eeb96-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnv8m\" (UniqueName: \"kubernetes.io/projected/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-kube-api-access-vnv8m\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.183983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.183933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.284310 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-sys\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.284310 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-metrics-client-ca\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.284554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.284554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.284554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.284554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpk6\" (UniqueName: \"kubernetes.io/projected/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-api-access-7jpk6\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.284554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-root\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.284943 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-textfile\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285034 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.284979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wjjl\" (UniqueName: \"kubernetes.io/projected/66b1d5ff-2546-4616-a72a-0e5dacdff116-kube-api-access-4wjjl\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61c89006-c0e2-46db-a8ab-16f7a63eeb96-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-accelerators-collector-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnv8m\" (UniqueName: \"kubernetes.io/projected/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-kube-api-access-vnv8m\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61c89006-c0e2-46db-a8ab-16f7a63eeb96-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-tls\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-wtmp\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.285684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.285441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/61c89006-c0e2-46db-a8ab-16f7a63eeb96-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.285684 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:17:24.285549    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 15 21:17:24.285684 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:17:24.285621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls podName:61c89006-c0e2-46db-a8ab-16f7a63eeb96 nodeName:}" failed. No retries permitted until 2026-09-15 21:17:24.785601603 +0000 UTC m=+73.186574831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-cddnv" (UID: "61c89006-c0e2-46db-a8ab-16f7a63eeb96") : secret "kube-state-metrics-tls" not found
Sep 15 21:17:24.286121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.286097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.287336 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.287317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.288040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.288016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.288131 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.288024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.302807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.302782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpk6\" (UniqueName: \"kubernetes.io/projected/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-api-access-7jpk6\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.331428 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.331388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnv8m\" (UniqueName: \"kubernetes.io/projected/a7ecd8f1-a2d3-414a-94eb-ea324edf341a-kube-api-access-vnv8m\") pod \"openshift-state-metrics-67dcf8c979-gh2cd\" (UID: \"a7ecd8f1-a2d3-414a-94eb-ea324edf341a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.386600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-wtmp\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-sys\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-metrics-client-ca\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-root\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-textfile\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-sys\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wjjl\" (UniqueName: \"kubernetes.io/projected/66b1d5ff-2546-4616-a72a-0e5dacdff116-kube-api-access-4wjjl\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-wtmp\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-accelerators-collector-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.386844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66b1d5ff-2546-4616-a72a-0e5dacdff116-root\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.387243 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.386859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-tls\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.387425 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.387386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-metrics-client-ca\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.387603 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.387566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-textfile\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.387966 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.387941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-accelerators-collector-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.389930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.389899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-tls\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.390036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.389949    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66b1d5ff-2546-4616-a72a-0e5dacdff116-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.401491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.401461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wjjl\" (UniqueName: \"kubernetes.io/projected/66b1d5ff-2546-4616-a72a-0e5dacdff116-kube-api-access-4wjjl\") pod \"node-exporter-ws9f2\" (UID: \"66b1d5ff-2546-4616-a72a-0e5dacdff116\") " pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.457606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.457574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"
Sep 15 21:17:24.488787 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.488740    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ws9f2"
Sep 15 21:17:24.510160 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:24.510132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b1d5ff_2546_4616_a72a_0e5dacdff116.slice/crio-70719ec42d7a0e8fa2f02cccaa095c045b53715b8a529285d6b334e14310e63d WatchSource:0}: Error finding container 70719ec42d7a0e8fa2f02cccaa095c045b53715b8a529285d6b334e14310e63d: Status 404 returned error can't find the container with id 70719ec42d7a0e8fa2f02cccaa095c045b53715b8a529285d6b334e14310e63d
Sep 15 21:17:24.759667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.759634    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd"]
Sep 15 21:17:24.789829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.789791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.792054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:24.792034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c89006-c0e2-46db-a8ab-16f7a63eeb96-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cddnv\" (UID: \"61c89006-c0e2-46db-a8ab-16f7a63eeb96\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:24.842587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:24.842557    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ecd8f1_a2d3_414a_94eb_ea324edf341a.slice/crio-80c688a7e30ebc90b64e28b10427bc34625547a262a4ce7d5f17605f41bb89b0 WatchSource:0}: Error finding container 80c688a7e30ebc90b64e28b10427bc34625547a262a4ce7d5f17605f41bb89b0: Status 404 returned error can't find the container with id 80c688a7e30ebc90b64e28b10427bc34625547a262a4ce7d5f17605f41bb89b0
Sep 15 21:17:25.067389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.067350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"
Sep 15 21:17:25.272272 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.272171    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:17:25.275149 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.275129    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.279534 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.279514    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:17:25.279640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.279515    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:17:25.280604 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.280567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:17:25.280760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.280602    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:17:25.280760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.280603    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:17:25.285424 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.285264    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:17:25.285424 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.285274    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:17:25.286742 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.285670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lsmkp\""
Sep 15 21:17:25.289168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.287845    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:17:25.291663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.291644    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:17:25.366170 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.366140    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv"]
Sep 15 21:17:25.370496 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:25.370450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c89006_c0e2_46db_a8ab_16f7a63eeb96.slice/crio-6180043f29a5cde24bb52545b0476eefc9dbc53bdac36fd1525e102424d668bd WatchSource:0}: Error finding container 6180043f29a5cde24bb52545b0476eefc9dbc53bdac36fd1525e102424d668bd: Status 404 returned error can't find the container with id 6180043f29a5cde24bb52545b0476eefc9dbc53bdac36fd1525e102424d668bd
Sep 15 21:17:25.395226 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgsj\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.395655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.395651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.401557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.401523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:17:25.478887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.478849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z4tbr" event={"ID":"7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94","Type":"ContainerStarted","Data":"d3aeb2ce8d7082906f71cd00ab24708cbe1e66a6d5d3ace52ec4b2d9b1565123"}
Sep 15 21:17:25.480111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.480084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv" event={"ID":"61c89006-c0e2-46db-a8ab-16f7a63eeb96","Type":"ContainerStarted","Data":"6180043f29a5cde24bb52545b0476eefc9dbc53bdac36fd1525e102424d668bd"}
Sep 15 21:17:25.481223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.481200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ws9f2" event={"ID":"66b1d5ff-2546-4616-a72a-0e5dacdff116","Type":"ContainerStarted","Data":"70719ec42d7a0e8fa2f02cccaa095c045b53715b8a529285d6b334e14310e63d"}
Sep 15 21:17:25.482928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.482907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd" event={"ID":"a7ecd8f1-a2d3-414a-94eb-ea324edf341a","Type":"ContainerStarted","Data":"148dd56c4ed94c889900782437c958c8b69f9e47ee956aba71f4f3171a1e99d0"}
Sep 15 21:17:25.483013 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.482948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd" event={"ID":"a7ecd8f1-a2d3-414a-94eb-ea324edf341a","Type":"ContainerStarted","Data":"82101c94556fb84a8c12362d25361214536ef80643f3b9d9c95be6bb50ea13ae"}
Sep 15 21:17:25.483013 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.482966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd" event={"ID":"a7ecd8f1-a2d3-414a-94eb-ea324edf341a","Type":"ContainerStarted","Data":"80c688a7e30ebc90b64e28b10427bc34625547a262a4ce7d5f17605f41bb89b0"}
Sep 15 21:17:25.496777 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.496888 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.496888 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.496888 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.496984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.497027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.497076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.497103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgsj\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.497323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.497132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.498513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.497813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.498513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.498452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.500473 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.500420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.500710 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.500682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.500919 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.500894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501028 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.500997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501433 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501433 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501433 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501574 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.501613 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.501584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.543617 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.543510    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z4tbr" podStartSLOduration=66.942725056 podStartE2EDuration="1m13.543489931s" totalImagesPullingTime="6.600764875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:25.539815821 +0000 UTC m=+73.940789066" watchObservedRunningTime="2026-09-15 21:17:25.543489931 +0000 UTC m=+73.944463180"
Sep 15 21:17:25.545975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.545950    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgsj\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj\") pod \"alertmanager-main-0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.584304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.584271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:17:25.866870 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:25.866784    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:17:26.065099 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:26.065062    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a88760_7dbf_400a_82ff_e5cb0d3242a0.slice/crio-42859f2c1e36e56038745acd68dda77e7b100a5e56e428bc6b9075f5934c5fa9 WatchSource:0}: Error finding container 42859f2c1e36e56038745acd68dda77e7b100a5e56e428bc6b9075f5934c5fa9: Status 404 returned error can't find the container with id 42859f2c1e36e56038745acd68dda77e7b100a5e56e428bc6b9075f5934c5fa9
Sep 15 21:17:26.487953 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:26.487909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"42859f2c1e36e56038745acd68dda77e7b100a5e56e428bc6b9075f5934c5fa9"}
Sep 15 21:17:26.490594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:26.490556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd" event={"ID":"a7ecd8f1-a2d3-414a-94eb-ea324edf341a","Type":"ContainerStarted","Data":"a01235a55c546009784b96488576d3d3f6731fef25ae37a57812b8ac8f3a2f1a"}
Sep 15 21:17:26.511853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:26.511790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-gh2cd" podStartSLOduration=1.445012806 podStartE2EDuration="2.511767843s" totalImagesPullingTime="1.066755037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:26.509934092 +0000 UTC m=+74.910907341" watchObservedRunningTime="2026-09-15 21:17:26.511767843 +0000 UTC m=+74.912741093"
Sep 15 21:17:27.151247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.151215    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"]
Sep 15 21:17:27.154231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.154207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.158297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.158275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 21:17:27.158923 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.158904    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 21:17:27.159040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.158934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 21:17:27.159040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.158939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 21:17:27.159375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.159350    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dnobnjnj9kp2d\""
Sep 15 21:17:27.159375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.159360    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pn96s\""
Sep 15 21:17:27.159547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.159413    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 21:17:27.176345 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.176316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"]
Sep 15 21:17:27.212842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.212799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.212842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.212845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213065 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.212880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213065 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.212963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-metrics-client-ca\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213065 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.213006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213065 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.213034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-grpc-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213219 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.213088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.213219 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.213128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q2j\" (UniqueName: \"kubernetes.io/projected/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-kube-api-access-42q2j\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313618 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-metrics-client-ca\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-grpc-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313757 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313757 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q2j\" (UniqueName: \"kubernetes.io/projected/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-kube-api-access-42q2j\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313865 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.313865 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.313827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.314438 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.314403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-metrics-client-ca\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.316389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.316354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.316494 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.316449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.316661 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.316634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.316739 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.316691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-grpc-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.316849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.316823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.317190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.317163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-secret-thanos-querier-tls\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.321628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.321609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q2j\" (UniqueName: \"kubernetes.io/projected/7ff809f2-20e9-45a2-bacb-b7e58f6d3a53-kube-api-access-42q2j\") pod \"thanos-querier-6b4974db4b-r7gx5\" (UID: \"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53\") " pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.464412 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.464357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:27.497267 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.497229    2578 generic.go:356] "Generic (PLEG): container finished" podID="66b1d5ff-2546-4616-a72a-0e5dacdff116" containerID="337bdf1396accdb0f802ba0e326fb6e62395987bca3037bca951160d2d7304d7" exitCode=0
Sep 15 21:17:27.497744 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.497281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ws9f2" event={"ID":"66b1d5ff-2546-4616-a72a-0e5dacdff116","Type":"ContainerDied","Data":"337bdf1396accdb0f802ba0e326fb6e62395987bca3037bca951160d2d7304d7"}
Sep 15 21:17:27.500005 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.499922    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064" exitCode=0
Sep 15 21:17:27.500356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.500329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064"}
Sep 15 21:17:27.753430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:27.753262    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"]
Sep 15 21:17:27.755798 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:27.755769    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff809f2_20e9_45a2_bacb_b7e58f6d3a53.slice/crio-42bcfaa9df697565e06c11a6b74804656a72f2e51db0d66bc87d19e924436f4c WatchSource:0}: Error finding container 42bcfaa9df697565e06c11a6b74804656a72f2e51db0d66bc87d19e924436f4c: Status 404 returned error can't find the container with id 42bcfaa9df697565e06c11a6b74804656a72f2e51db0d66bc87d19e924436f4c
Sep 15 21:17:28.507928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.507891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ws9f2" event={"ID":"66b1d5ff-2546-4616-a72a-0e5dacdff116","Type":"ContainerStarted","Data":"87eb0f58bdb96830a37e386549e1d488dcb1f14c926b52e1d3d62c536da0b662"}
Sep 15 21:17:28.508402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.507959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ws9f2" event={"ID":"66b1d5ff-2546-4616-a72a-0e5dacdff116","Type":"ContainerStarted","Data":"1c9d69c4cd53e1148f26b45bcc02c11114adbb2460f60377636ae55f51ccaef3"}
Sep 15 21:17:28.509497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.509466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"42bcfaa9df697565e06c11a6b74804656a72f2e51db0d66bc87d19e924436f4c"}
Sep 15 21:17:28.511883 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.511847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv" event={"ID":"61c89006-c0e2-46db-a8ab-16f7a63eeb96","Type":"ContainerStarted","Data":"01aa7528b688c2695d6c035e9c6242dcb19a8ac563301f7a2a3da943b6a82976"}
Sep 15 21:17:28.511988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.511897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv" event={"ID":"61c89006-c0e2-46db-a8ab-16f7a63eeb96","Type":"ContainerStarted","Data":"e5044c49c867706bead2e1fc5caa7cbfe643d45e51629e9a8f8c6fdb3f6d86f8"}
Sep 15 21:17:28.511988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.511928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv" event={"ID":"61c89006-c0e2-46db-a8ab-16f7a63eeb96","Type":"ContainerStarted","Data":"c0e7acffd471472bf307e56a14f35293c73f4aaf82f08bf22c54c2dc1255d817"}
Sep 15 21:17:28.527671 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.527598    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ws9f2" podStartSLOduration=2.411120205 podStartE2EDuration="4.527581184s" totalImagesPullingTime="2.053192657s" totalInitContainerRuntime="63.268322ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:28.52718802 +0000 UTC m=+76.928161289" watchObservedRunningTime="2026-09-15 21:17:28.527581184 +0000 UTC m=+76.928554432"
Sep 15 21:17:28.544973 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.544917    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cddnv" podStartSLOduration=1.7860393399999999 podStartE2EDuration="4.544900676s" totalImagesPullingTime="2.758861336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:28.544315046 +0000 UTC m=+76.945288295" watchObservedRunningTime="2026-09-15 21:17:28.544900676 +0000 UTC m=+76.945873954"
Sep 15 21:17:28.695023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.694859    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-fb4bc949b-nr7xr"]
Sep 15 21:17:28.699578 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.699549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.704829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704420    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9l0arnpu1ju1o\""
Sep 15 21:17:28.704829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gc9zd\""
Sep 15 21:17:28.704829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 21:17:28.704829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704493    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 21:17:28.704829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704826    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 21:17:28.705161 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.704940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 21:17:28.717693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.717674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fb4bc949b-nr7xr"]
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-metrics-server-audit-profiles\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68rn\" (UniqueName: \"kubernetes.io/projected/978ff269-e496-4abd-b062-560e64a733df-kube-api-access-b68rn\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-tls\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-client-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/978ff269-e496-4abd-b062-560e64a733df-audit-log\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.831983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.831762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-client-certs\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.882261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.882228    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"]
Sep 15 21:17:28.886614 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.886587    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:28.888745 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.888700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 21:17:28.888996 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.888935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lkkbl\""
Sep 15 21:17:28.894117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.894095    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"]
Sep 15 21:17:28.932542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68rn\" (UniqueName: \"kubernetes.io/projected/978ff269-e496-4abd-b062-560e64a733df-kube-api-access-b68rn\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.932542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-tls\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-client-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksnh\" (UniqueName: \"kubernetes.io/projected/ccdb529a-a8f1-42a8-82ac-6568c0387667-kube-api-access-pksnh\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ccdb529a-a8f1-42a8-82ac-6568c0387667-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/978ff269-e496-4abd-b062-560e64a733df-audit-log\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.932825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-client-certs\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.933117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.932905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-metrics-server-audit-profiles\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.933330 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.933282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/978ff269-e496-4abd-b062-560e64a733df-audit-log\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.933672 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.933630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.934073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.934034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/978ff269-e496-4abd-b062-560e64a733df-metrics-server-audit-profiles\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.935862 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.935838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-tls\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.935964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.935905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-secret-metrics-server-client-certs\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.935964 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.935938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/978ff269-e496-4abd-b062-560e64a733df-client-ca-bundle\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:28.939727 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:28.939692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68rn\" (UniqueName: \"kubernetes.io/projected/978ff269-e496-4abd-b062-560e64a733df-kube-api-access-b68rn\") pod \"metrics-server-fb4bc949b-nr7xr\" (UID: \"978ff269-e496-4abd-b062-560e64a733df\") " pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:29.011627 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.011593    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:29.033740 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.033685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksnh\" (UniqueName: \"kubernetes.io/projected/ccdb529a-a8f1-42a8-82ac-6568c0387667-kube-api-access-pksnh\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:29.033887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.033804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ccdb529a-a8f1-42a8-82ac-6568c0387667-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:29.037328 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.037297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ccdb529a-a8f1-42a8-82ac-6568c0387667-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:29.042534 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.042487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksnh\" (UniqueName: \"kubernetes.io/projected/ccdb529a-a8f1-42a8-82ac-6568c0387667-kube-api-access-pksnh\") pod \"monitoring-plugin-5d85778b9d-kw422\" (UID: \"ccdb529a-a8f1-42a8-82ac-6568c0387667\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:29.200116 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.200078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:29.298078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.298038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fb4bc949b-nr7xr"]
Sep 15 21:17:29.414537 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:29.414503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978ff269_e496_4abd_b062_560e64a733df.slice/crio-4095d683c4083a2db685977e53b713722b9bc98018b18866fad0c0c2f6f3a2dc WatchSource:0}: Error finding container 4095d683c4083a2db685977e53b713722b9bc98018b18866fad0c0c2f6f3a2dc: Status 404 returned error can't find the container with id 4095d683c4083a2db685977e53b713722b9bc98018b18866fad0c0c2f6f3a2dc
Sep 15 21:17:29.516997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.516962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr" event={"ID":"978ff269-e496-4abd-b062-560e64a733df","Type":"ContainerStarted","Data":"4095d683c4083a2db685977e53b713722b9bc98018b18866fad0c0c2f6f3a2dc"}
Sep 15 21:17:29.682476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:29.682433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"]
Sep 15 21:17:30.091692 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:30.091669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdb529a_a8f1_42a8_82ac_6568c0387667.slice/crio-e5ceb7c04ef2db0213846dc246e5ed2335e99e0c03fc3c2295af4a825a4daf64 WatchSource:0}: Error finding container e5ceb7c04ef2db0213846dc246e5ed2335e99e0c03fc3c2295af4a825a4daf64: Status 404 returned error can't find the container with id e5ceb7c04ef2db0213846dc246e5ed2335e99e0c03fc3c2295af4a825a4daf64
Sep 15 21:17:30.410769 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.410739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ddc8578db-7l7rv"
Sep 15 21:17:30.451764 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.449634    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:17:30.462272 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.462232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.467133 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.467104    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xtxnm\""
Sep 15 21:17:30.467542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.467519    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:17:30.467938 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.467917    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:17:30.468325 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.468304    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:17:30.468467 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.468317    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:17:30.468582 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.468363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:17:30.468894 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.468877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:17:30.469040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.468962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pbnlssha5v31\""
Sep 15 21:17:30.469245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.469221    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:17:30.469328 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.469250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:17:30.469452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.469416    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:17:30.469969 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.469895    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:17:30.472057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.472038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:17:30.473042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.473024    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:17:30.473781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.473303    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:17:30.479338 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.479154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:17:30.479574 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.479501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:17:30.531443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.531387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"}
Sep 15 21:17:30.531925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.531526    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"}
Sep 15 21:17:30.531925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.531575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"}
Sep 15 21:17:30.531925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.531614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"}
Sep 15 21:17:30.531925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.531654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"}
Sep 15 21:17:30.535579 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.535545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"1b1ce68640ae0326fab8fee4e9d2070cc0fba061e444ece32c8572884ad75aec"}
Sep 15 21:17:30.535735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.535630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"2ac7ab00cff9eafecde77163958570b38be803dd339b1b20da7bd6e71227e80e"}
Sep 15 21:17:30.535735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.535676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"40a7fe0d589bb3e3d4db5a6ec14888d7a4018ad720243ffffc9c7f43c0106fb7"}
Sep 15 21:17:30.537183 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.537158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422" event={"ID":"ccdb529a-a8f1-42a8-82ac-6568c0387667","Type":"ContainerStarted","Data":"e5ceb7c04ef2db0213846dc246e5ed2335e99e0c03fc3c2295af4a825a4daf64"}
Sep 15 21:17:30.548892 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.548846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.548931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.548958    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.548991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549306    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549671 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549671 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6fj\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549862    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.549997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.549895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.650987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6fj\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.651860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.652806 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.652806 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.652806 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.651594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.652806 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.652471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.652806 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.652630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.654368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.653436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.654368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.654092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.656150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.656122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.656337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.656309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.657152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.657177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.659251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.659419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.660446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.660894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.661024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.661205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.661508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.661869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.661792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.665891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.663318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.665891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.663705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.666279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.666235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.673222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.673194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6fj\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj\") pod \"prometheus-k8s-0\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:30.781782 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:30.781741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:31.103570 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:31.103406    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:17:31.810292 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:17:31.810263    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d86865_26b7_44ca_83ba_f537225c8645.slice/crio-b3e59948f2374a796089876fa82ca6c03c36ad68a75da10464a48bd0ecdcf03b WatchSource:0}: Error finding container b3e59948f2374a796089876fa82ca6c03c36ad68a75da10464a48bd0ecdcf03b: Status 404 returned error can't find the container with id b3e59948f2374a796089876fa82ca6c03c36ad68a75da10464a48bd0ecdcf03b
Sep 15 21:17:32.549548 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.549506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerStarted","Data":"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"}
Sep 15 21:17:32.552683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.552650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"74e89df97705d5cbf694117e280cfadda38975a1c9628872f5b6ed35070b9686"}
Sep 15 21:17:32.552836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.552738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"29862240cd282fcd74d393d7d15bb5792ae147005a68270ab1bf8f563579c87e"}
Sep 15 21:17:32.552836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.552794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:32.552836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.552829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" event={"ID":"7ff809f2-20e9-45a2-bacb-b7e58f6d3a53","Type":"ContainerStarted","Data":"c4ff04b9e368772c5359238bcd63cd064b794b38c3d50857acb67960439a938f"}
Sep 15 21:17:32.554442 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.554414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr" event={"ID":"978ff269-e496-4abd-b062-560e64a733df","Type":"ContainerStarted","Data":"232d7a414fc0c8ee70f6bff7f8b5e71eb52fae69b1f8e50196705404ef5d314f"}
Sep 15 21:17:32.556047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.556023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422" event={"ID":"ccdb529a-a8f1-42a8-82ac-6568c0387667","Type":"ContainerStarted","Data":"166b12a117403d37aedc965ef1232b37055a3f9706cbaf51d0f186b998ed91ad"}
Sep 15 21:17:32.556256 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.556230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:32.557666 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.557642    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057" exitCode=0
Sep 15 21:17:32.557765 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.557676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"}
Sep 15 21:17:32.557765 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.557748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"b3e59948f2374a796089876fa82ca6c03c36ad68a75da10464a48bd0ecdcf03b"}
Sep 15 21:17:32.562069 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.562051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422"
Sep 15 21:17:32.582637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.582593    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.774652259 podStartE2EDuration="7.582580587s" totalImagesPullingTime="4.791042246s" totalInitContainerRuntime="16.886082ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:32.580697027 +0000 UTC m=+80.981670285" watchObservedRunningTime="2026-09-15 21:17:32.582580587 +0000 UTC m=+80.983553833"
Sep 15 21:17:32.604844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.604775    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr" podStartSLOduration=2.117544864 podStartE2EDuration="4.604755261s" totalImagesPullingTime="2.487210397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:32.604359612 +0000 UTC m=+81.005332859" watchObservedRunningTime="2026-09-15 21:17:32.604755261 +0000 UTC m=+81.005728501"
Sep 15 21:17:32.623637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.623583    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kw422" podStartSLOduration=2.8144150310000002 podStartE2EDuration="4.623567962s" totalImagesPullingTime="1.809152931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:32.622766054 +0000 UTC m=+81.023739300" watchObservedRunningTime="2026-09-15 21:17:32.623567962 +0000 UTC m=+81.024541198"
Sep 15 21:17:32.682234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:32.682174    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5" podStartSLOduration=1.777413228 podStartE2EDuration="5.682149796s" totalImagesPullingTime="3.904736568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:32.679691129 +0000 UTC m=+81.080664374" watchObservedRunningTime="2026-09-15 21:17:32.682149796 +0000 UTC m=+81.083123043"
Sep 15 21:17:36.584278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"}
Sep 15 21:17:36.584704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"}
Sep 15 21:17:36.584704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"}
Sep 15 21:17:36.584704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"}
Sep 15 21:17:36.584704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"}
Sep 15 21:17:36.584704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.584416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerStarted","Data":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"}
Sep 15 21:17:36.662838 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:36.660897    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.467665017 podStartE2EDuration="6.660874177s" totalImagesPullingTime="3.16214834s" totalInitContainerRuntime="31.06082ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:36.659800832 +0000 UTC m=+85.060774079" watchObservedRunningTime="2026-09-15 21:17:36.660874177 +0000 UTC m=+85.061847427"
Sep 15 21:17:38.572607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:38.572580    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b4974db4b-r7gx5"
Sep 15 21:17:40.782118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:40.782074    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:17:49.012305 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:49.012268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:49.012688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:49.012334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:17:53.474697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:17:53.474661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pd7zt"
Sep 15 21:18:09.017635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:09.017601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:18:09.021456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:09.021435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-fb4bc949b-nr7xr"
Sep 15 21:18:30.782559 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:30.782518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:30.802121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:30.802094    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:30.818095 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:30.818070    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:44.565359 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:44.565918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565622    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="alertmanager" containerID="cri-o://3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64" gracePeriod=120
Sep 15 21:18:44.565918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565752    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="prom-label-proxy" containerID="cri-o://eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e" gracePeriod=120
Sep 15 21:18:44.565918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565808    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="config-reloader" containerID="cri-o://1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582" gracePeriod=120
Sep 15 21:18:44.565918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565768    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-web" containerID="cri-o://da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5" gracePeriod=120
Sep 15 21:18:44.565918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565822    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy" containerID="cri-o://bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375" gracePeriod=120
Sep 15 21:18:44.566202 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.565749    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-metric" containerID="cri-o://4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d" gracePeriod=120
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861672    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e" exitCode=0
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861699    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375" exitCode=0
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861708    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5" exitCode=0
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861732    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582" exitCode=0
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861741    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64" exitCode=0
Sep 15 21:18:44.861786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"}
Sep 15 21:18:44.862068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"}
Sep 15 21:18:44.862068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"}
Sep 15 21:18:44.862068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"}
Sep 15 21:18:44.862068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:44.861905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"}
Sep 15 21:18:45.811043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.811007    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:45.872066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.872034    2578 generic.go:356] "Generic (PLEG): container finished" podID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerID="4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d" exitCode=0
Sep 15 21:18:45.872235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.872114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"}
Sep 15 21:18:45.872235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.872152    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:45.872235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.872185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35a88760-7dbf-400a-82ff-e5cb0d3242a0","Type":"ContainerDied","Data":"42859f2c1e36e56038745acd68dda77e7b100a5e56e428bc6b9075f5934c5fa9"}
Sep 15 21:18:45.872235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.872220    2578 scope.go:117] "RemoveContainer" containerID="eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"
Sep 15 21:18:45.882926 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.880872    2578 scope.go:117] "RemoveContainer" containerID="4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"
Sep 15 21:18:45.889629 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.889608    2578 scope.go:117] "RemoveContainer" containerID="bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"
Sep 15 21:18:45.896934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.896915    2578 scope.go:117] "RemoveContainer" containerID="da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"
Sep 15 21:18:45.905822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.905803    2578 scope.go:117] "RemoveContainer" containerID="1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"
Sep 15 21:18:45.908027 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908136 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908136 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908136 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908295 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908295 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908612 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908409    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908612 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:45.908832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.908832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmgsj\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.909188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.909188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.908921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls\") pod \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\" (UID: \"35a88760-7dbf-400a-82ff-e5cb0d3242a0\") "
Sep 15 21:18:45.909304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.909204    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-main-db\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:45.909931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.909629    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:45.910300 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.910230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:45.912640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.912564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out" (OuterVolumeSpecName: "config-out") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:45.912640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.912598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:45.913274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.913235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume" (OuterVolumeSpecName: "config-volume") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.914413 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.914385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.914641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.914617    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj" (OuterVolumeSpecName: "kube-api-access-nmgsj") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "kube-api-access-nmgsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:45.914807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.914784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.914887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.914863    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.915096 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.915070    2578 scope.go:117] "RemoveContainer" containerID="3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"
Sep 15 21:18:45.915928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.915869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.917485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.917459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.923750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.923712    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config" (OuterVolumeSpecName: "web-config") pod "35a88760-7dbf-400a-82ff-e5cb0d3242a0" (UID: "35a88760-7dbf-400a-82ff-e5cb0d3242a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:45.932867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.932848    2578 scope.go:117] "RemoveContainer" containerID="3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064"
Sep 15 21:18:45.939930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.939906    2578 scope.go:117] "RemoveContainer" containerID="eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"
Sep 15 21:18:45.940221 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.940200    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e\": container with ID starting with eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e not found: ID does not exist" containerID="eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"
Sep 15 21:18:45.940290 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940229    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e"} err="failed to get container status \"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e\": rpc error: code = NotFound desc = could not find container \"eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e\": container with ID starting with eaad21c8adfcc5dafecc604993c744dbd68f95b134cde5f8eb933c131423e34e not found: ID does not exist"
Sep 15 21:18:45.940290 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940259    2578 scope.go:117] "RemoveContainer" containerID="4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"
Sep 15 21:18:45.940511 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.940494    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d\": container with ID starting with 4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d not found: ID does not exist" containerID="4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"
Sep 15 21:18:45.940547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940520    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d"} err="failed to get container status \"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d\": rpc error: code = NotFound desc = could not find container \"4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d\": container with ID starting with 4c36a6a994f9673d49e11af88e410063aa7f78878e261c0e91397e55eb17258d not found: ID does not exist"
Sep 15 21:18:45.940547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940541    2578 scope.go:117] "RemoveContainer" containerID="bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"
Sep 15 21:18:45.940785 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.940767    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375\": container with ID starting with bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375 not found: ID does not exist" containerID="bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"
Sep 15 21:18:45.940849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940790    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375"} err="failed to get container status \"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375\": rpc error: code = NotFound desc = could not find container \"bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375\": container with ID starting with bc6ef8bc40b4b0eebc147ea71960530a696a8f82c2d990adb6df37bb34be7375 not found: ID does not exist"
Sep 15 21:18:45.940849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.940804    2578 scope.go:117] "RemoveContainer" containerID="da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"
Sep 15 21:18:45.941017 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.941000    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5\": container with ID starting with da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5 not found: ID does not exist" containerID="da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"
Sep 15 21:18:45.941057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5"} err="failed to get container status \"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5\": rpc error: code = NotFound desc = could not find container \"da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5\": container with ID starting with da28fd3126696ca97bf6641af67a7478de930363bd8fe5bf0b944cdc85168fd5 not found: ID does not exist"
Sep 15 21:18:45.941057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941034    2578 scope.go:117] "RemoveContainer" containerID="1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"
Sep 15 21:18:45.941243 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.941227    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582\": container with ID starting with 1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582 not found: ID does not exist" containerID="1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"
Sep 15 21:18:45.941282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941247    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582"} err="failed to get container status \"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582\": rpc error: code = NotFound desc = could not find container \"1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582\": container with ID starting with 1dbfcfecd0335f3981e7df35ac93a42b8e992d550ea4c82b9d09675f43f8d582 not found: ID does not exist"
Sep 15 21:18:45.941282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941258    2578 scope.go:117] "RemoveContainer" containerID="3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"
Sep 15 21:18:45.941442 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.941427    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64\": container with ID starting with 3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64 not found: ID does not exist" containerID="3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"
Sep 15 21:18:45.941481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941444    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64"} err="failed to get container status \"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64\": rpc error: code = NotFound desc = could not find container \"3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64\": container with ID starting with 3af3f420d6c625e16b5f87c1f4f7cfaf35c0ad2f5e1898de251e71c9fa51ee64 not found: ID does not exist"
Sep 15 21:18:45.941481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941456    2578 scope.go:117] "RemoveContainer" containerID="3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064"
Sep 15 21:18:45.941639 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:45.941624    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064\": container with ID starting with 3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064 not found: ID does not exist" containerID="3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064"
Sep 15 21:18:45.941676 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:45.941642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064"} err="failed to get container status \"3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064\": rpc error: code = NotFound desc = could not find container \"3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064\": container with ID starting with 3143520d931e0a7cd7a9f901d0cd82734b63c8eb5cd43a3723dc14889eb2a064 not found: ID does not exist"
Sep 15 21:18:46.010484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010456    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-main-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010478    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010490    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-cluster-tls-config\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010501    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-web-config\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010515    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010524    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010532    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-tls-assets\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010540    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010549    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35a88760-7dbf-400a-82ff-e5cb0d3242a0-metrics-client-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010557    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35a88760-7dbf-400a-82ff-e5cb0d3242a0-config-out\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010567    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgsj\" (UniqueName: \"kubernetes.io/projected/35a88760-7dbf-400a-82ff-e5cb0d3242a0-kube-api-access-nmgsj\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.010684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.010575    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35a88760-7dbf-400a-82ff-e5cb0d3242a0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:46.195658 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.195629    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:46.203608 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.203578    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:46.245327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245298    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:46.245599 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245587    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="init-config-reloader"
Sep 15 21:18:46.245637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245601    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="init-config-reloader"
Sep 15 21:18:46.245637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245612    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-web"
Sep 15 21:18:46.245637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245618    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-web"
Sep 15 21:18:46.245637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245625    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="prom-label-proxy"
Sep 15 21:18:46.245637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245630    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="prom-label-proxy"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245643    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245649    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245655    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-metric"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-metric"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="alertmanager"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245675    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="alertmanager"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245685    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="config-reloader"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245690    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="config-reloader"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245748    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="alertmanager"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245756    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="prom-label-proxy"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-metric"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="config-reloader"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245773    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy-web"
Sep 15 21:18:46.245808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.245778    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" containerName="kube-rbac-proxy"
Sep 15 21:18:46.249144 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.249126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.254040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.253982    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:18:46.254040 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.253998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lsmkp\""
Sep 15 21:18:46.254222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.253998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:18:46.254222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:18:46.254222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:18:46.254222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254193    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:18:46.254486 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254471    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:18:46.254589 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:18:46.254701 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.254686    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:18:46.259096 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.259077    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:18:46.271754 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.271729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:46.313451 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-out\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-web-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.313990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.313898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzmg\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-kube-api-access-lnzmg\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.414872 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzmg\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-kube-api-access-lnzmg\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414880    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-out\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.414980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415309 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415309 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415309 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415297    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415453 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415453 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415453 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415605 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-web-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.415972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.415877    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.416087 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.416058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8468bb53-031f-4260-86f5-7ac8e3974f5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418007 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.417978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-out\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418137 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.418112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418207 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.418176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.418373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-web-config\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418504 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.418423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.418606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.418588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.419028 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.419006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.419104 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.419073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.419926 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.419908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8468bb53-031f-4260-86f5-7ac8e3974f5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.422902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.422886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzmg\" (UniqueName: \"kubernetes.io/projected/8468bb53-031f-4260-86f5-7ac8e3974f5f-kube-api-access-lnzmg\") pod \"alertmanager-main-0\" (UID: \"8468bb53-031f-4260-86f5-7ac8e3974f5f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.559164 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.559078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:18:46.846973 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.846897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:18:46.849873 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:18:46.849834    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8468bb53_031f_4260_86f5_7ac8e3974f5f.slice/crio-ddea296432082988d19c0adfa60e79e000e0d44e5b17760fe3a5799f2a173d67 WatchSource:0}: Error finding container ddea296432082988d19c0adfa60e79e000e0d44e5b17760fe3a5799f2a173d67: Status 404 returned error can't find the container with id ddea296432082988d19c0adfa60e79e000e0d44e5b17760fe3a5799f2a173d67
Sep 15 21:18:46.878712 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:46.878683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"ddea296432082988d19c0adfa60e79e000e0d44e5b17760fe3a5799f2a173d67"}
Sep 15 21:18:47.884619 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:47.884585    2578 generic.go:356] "Generic (PLEG): container finished" podID="8468bb53-031f-4260-86f5-7ac8e3974f5f" containerID="27f7368a6adab1fc50b8b3677509bbda7c31cdff1222e6e8863b436286de2b38" exitCode=0
Sep 15 21:18:47.884995 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:47.884655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerDied","Data":"27f7368a6adab1fc50b8b3677509bbda7c31cdff1222e6e8863b436286de2b38"}
Sep 15 21:18:48.075642 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.075613    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a88760-7dbf-400a-82ff-e5cb0d3242a0" path="/var/lib/kubelet/pods/35a88760-7dbf-400a-82ff-e5cb0d3242a0/volumes"
Sep 15 21:18:48.682917 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.682861    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5699449f55-t62h8"]
Sep 15 21:18:48.685675 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.685645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.688709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-k6j4s\""
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.688701    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.689053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.688709    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.689332    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 21:18:48.691752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.691039    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 21:18:48.698591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.698557    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 21:18:48.705986 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.705959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5699449f55-t62h8"]
Sep 15 21:18:48.736993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.736956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-telemeter-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.736993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.736991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnch\" (UniqueName: \"kubernetes.io/projected/cade5465-0375-4250-9dac-31e5503d0453-kube-api-access-brnch\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737216 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-metrics-client-ca\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737216 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737216 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737216 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-federate-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737388 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.737388 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.737260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-serving-certs-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.837992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.837946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-metrics-client-ca\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-federate-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-serving-certs-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838437 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-telemeter-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838437 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnch\" (UniqueName: \"kubernetes.io/projected/cade5465-0375-4250-9dac-31e5503d0453-kube-api-access-brnch\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.838957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-serving-certs-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.839167 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.838966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-metrics-client-ca\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.839360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.839338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cade5465-0375-4250-9dac-31e5503d0453-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.841276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.841248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-telemeter-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.841379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.841248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-federate-client-tls\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.841379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.841351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.841495 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.841384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cade5465-0375-4250-9dac-31e5503d0453-secret-telemeter-client\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.846765 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.846739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnch\" (UniqueName: \"kubernetes.io/projected/cade5465-0375-4250-9dac-31e5503d0453-kube-api-access-brnch\") pod \"telemeter-client-5699449f55-t62h8\" (UID: \"cade5465-0375-4250-9dac-31e5503d0453\") " pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:48.892627 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"9f81dfc3ffc3174ade9fd0ddfec56916f18335bf7a29fed45c0a9ddb813cfca0"}
Sep 15 21:18:48.893014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"0fa7132e1e446185dcb37b38970753396115923bfca869627f2ac13e63440d5f"}
Sep 15 21:18:48.893014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"76bf9fd2122da53cbd0013a819c7ff870ab908cea7013fabbc98cbc7090f0762"}
Sep 15 21:18:48.893014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"6534d7a09b72f3c98c77ed02dca0442116d3e75f0e519824432413640638f347"}
Sep 15 21:18:48.893014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"1a956512a3b599bd0ef50528514eb3e15392d2e62f2e6a92241f2592f8c2d40e"}
Sep 15 21:18:48.893014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.892770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8468bb53-031f-4260-86f5-7ac8e3974f5f","Type":"ContainerStarted","Data":"efddbf65cde2be80c36fb0805bfab06a8120e6b26935d1ffbea1f6557820302c"}
Sep 15 21:18:48.924791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.924740    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.909480475 podStartE2EDuration="2.924706192s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.225717ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:48.923165938 +0000 UTC m=+157.324139184" watchObservedRunningTime="2026-09-15 21:18:48.924706192 +0000 UTC m=+157.325679440"
Sep 15 21:18:48.973067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973034    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:48.973482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973378    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="prometheus" containerID="cri-o://90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45" gracePeriod=600
Sep 15 21:18:48.973482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973414    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="thanos-sidecar" containerID="cri-o://b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a" gracePeriod=600
Sep 15 21:18:48.973482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973455    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-thanos" containerID="cri-o://064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e" gracePeriod=600
Sep 15 21:18:48.973482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973455    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="config-reloader" containerID="cri-o://727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a" gracePeriod=600
Sep 15 21:18:48.973759 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy" containerID="cri-o://6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04" gracePeriod=600
Sep 15 21:18:48.973759 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:48.973459    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-web" containerID="cri-o://280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb" gracePeriod=600
Sep 15 21:18:49.001498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.001468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8"
Sep 15 21:18:49.221074 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.219044    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:49.241394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241502    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241609    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241637    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241777    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241810    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241813    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241875    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.241982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.242358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.241991    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.242358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.242358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb6fj\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.242358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle\") pod \"a9d86865-26b7-44ca-83ba-f537225c8645\" (UID: \"a9d86865-26b7-44ca-83ba-f537225c8645\") "
Sep 15 21:18:49.242358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242123    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.242593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242466    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.242593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.242485    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.243223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.243192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.243610 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.243563    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.243709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.243666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.244638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.243968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.244638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.244278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:18:49.244638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.244391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:49.244638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.244391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.247112 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.247595 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:49.247680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.247929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.247929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.247929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.247884    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.248270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.248227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj" (OuterVolumeSpecName: "kube-api-access-kb6fj") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "kube-api-access-kb6fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:18:49.248696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.248584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out" (OuterVolumeSpecName: "config-out") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:18:49.248696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.248670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config" (OuterVolumeSpecName: "config") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.248696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.248687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.260829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.260798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config" (OuterVolumeSpecName: "web-config") pod "a9d86865-26b7-44ca-83ba-f537225c8645" (UID: "a9d86865-26b7-44ca-83ba-f537225c8645"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:18:49.308140 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.308104    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5699449f55-t62h8"]
Sep 15 21:18:49.313301 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:18:49.313272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcade5465_0375_4250_9dac_31e5503d0453.slice/crio-4092ddce0ae39893ee8d73179f569ec9debb8b0e8af39bfa4b1fb69e9466b8f1 WatchSource:0}: Error finding container 4092ddce0ae39893ee8d73179f569ec9debb8b0e8af39bfa4b1fb69e9466b8f1: Status 404 returned error can't find the container with id 4092ddce0ae39893ee8d73179f569ec9debb8b0e8af39bfa4b1fb69e9466b8f1
Sep 15 21:18:49.343394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343361    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-metrics-client-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343397    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343414    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343431    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-kube-rbac-proxy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343447    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-db\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343461    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-metrics-client-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343476    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-web-config\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343492    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb6fj\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-kube-api-access-kb6fj\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343504    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343517    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-config\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343526    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343535    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d86865-26b7-44ca-83ba-f537225c8645-config-out\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343545    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343553    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d86865-26b7-44ca-83ba-f537225c8645-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343561    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d86865-26b7-44ca-83ba-f537225c8645-tls-assets\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343570    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343581    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-grpc-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.343587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.343590    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d86865-26b7-44ca-83ba-f537225c8645-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:18:49.898128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.898089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8" event={"ID":"cade5465-0375-4250-9dac-31e5503d0453","Type":"ContainerStarted","Data":"4092ddce0ae39893ee8d73179f569ec9debb8b0e8af39bfa4b1fb69e9466b8f1"}
Sep 15 21:18:49.901275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901249    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e" exitCode=0
Sep 15 21:18:49.901275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901271    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04" exitCode=0
Sep 15 21:18:49.901275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901278    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb" exitCode=0
Sep 15 21:18:49.901275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901283    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a" exitCode=0
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901288    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a" exitCode=0
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901293    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9d86865-26b7-44ca-83ba-f537225c8645" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45" exitCode=0
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"}
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901387    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"}
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901330    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"}
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"}
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"}
Sep 15 21:18:49.901546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901485    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.901984 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"}
Sep 15 21:18:49.901984 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.901622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9d86865-26b7-44ca-83ba-f537225c8645","Type":"ContainerDied","Data":"b3e59948f2374a796089876fa82ca6c03c36ad68a75da10464a48bd0ecdcf03b"}
Sep 15 21:18:49.909708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.909587    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.916651 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.916628    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.923659 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.923637    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.925386 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.925360    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:49.930545 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.930523    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:49.931242 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.931223    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.937997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.937978    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.947339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.947304    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.954693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.954673    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.954994 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.954967    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.955052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.954994    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.955052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955011    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.955221 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.955206    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.955261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955224    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.955261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955236    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.955470 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.955451    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.955510 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.955510 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955499    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.955827 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.955697    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.955827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955737    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.955827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955750    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.955999 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.955948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.955999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.955999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.955979    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.956181 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.956166    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.956217 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956183    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.956217 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956195    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.956498 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:18:49.956446    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.956498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956470    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.956498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956484    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.956883 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956750    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.956883 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.956774    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.957121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.957095    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.957193 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.957125    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.957535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.957469    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.957535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.957494    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.958204 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958181    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.958333 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958313    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.958628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958605    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.958628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958624    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.958920 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958902    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:49.958989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.959030 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.958992    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.959238 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959224    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-web"
Sep 15 21:18:49.959302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959239    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-web"
Sep 15 21:18:49.959302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959240    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.959302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959266    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="prometheus"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959332    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="prometheus"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="config-reloader"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959365    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="config-reloader"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959379    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959388    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959413    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="init-config-reloader"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959422    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="init-config-reloader"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959432    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="thanos-sidecar"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="thanos-sidecar"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-thanos"
Sep 15 21:18:49.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959458    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-thanos"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959523    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959538    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="config-reloader"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959579    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="prometheus"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959589    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-web"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959599    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy-thanos"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959609    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="thanos-sidecar"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959619    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" containerName="kube-rbac-proxy"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959755    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959770    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.960035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.959993    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.960615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.960615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960185    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.960615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960372    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.960615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960393    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.960885 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960690    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.960885 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.960731    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.961044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.961044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961043    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.961288 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961268    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.961288 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961286    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.961516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961498    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.961557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961517    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.961792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961773    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.961847 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.961796    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.962094 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.962150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962095    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.962353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.962389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962353    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.962584 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962563    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.962660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962585    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.962920 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962902    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.962966 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.962920    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.963172 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.963172 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963171    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.963501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963462    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.963501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963487    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.963785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963760    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.963785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.963786    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.964068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.964033    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.964159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.964071    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.965196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.965149    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.965196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.965191    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.966150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.965397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:49.966297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.966255    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.966297 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.966295    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.967738 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.967677    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.967738 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.967708    2578 scope.go:117] "RemoveContainer" containerID="064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"
Sep 15 21:18:49.968286 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968262    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e"} err="failed to get container status \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": rpc error: code = NotFound desc = could not find container \"064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e\": container with ID starting with 064e753746861267b242a1c2d29778e1b18b0c4c39dd8d2b9289a1434b370b1e not found: ID does not exist"
Sep 15 21:18:49.968356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968289    2578 scope.go:117] "RemoveContainer" containerID="6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"
Sep 15 21:18:49.968409 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:18:49.968638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:18:49.968735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968682    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04"} err="failed to get container status \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": rpc error: code = NotFound desc = could not find container \"6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04\": container with ID starting with 6e93f3457299fd73988cbc498b15587139a84243ce605994beab230115273b04 not found: ID does not exist"
Sep 15 21:18:49.968735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968711    2578 scope.go:117] "RemoveContainer" containerID="280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"
Sep 15 21:18:49.968867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968798    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:18:49.968867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.968806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:18:49.969259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.969118    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:18:49.969259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.969220    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb"} err="failed to get container status \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": rpc error: code = NotFound desc = could not find container \"280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb\": container with ID starting with 280cc120c69074d42e2670b228a42d852dc27dd089b1f1bea0b484a4121d88eb not found: ID does not exist"
Sep 15 21:18:49.969259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.969244    2578 scope.go:117] "RemoveContainer" containerID="b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"
Sep 15 21:18:49.970709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xtxnm\""
Sep 15 21:18:49.970709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970464    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a"} err="failed to get container status \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": rpc error: code = NotFound desc = could not find container \"b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a\": container with ID starting with b3f04281a3788f6613d04abfc3c27feb47a5d752bf6b95b7ca6b3871d12c5f3a not found: ID does not exist"
Sep 15 21:18:49.970709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970490    2578 scope.go:117] "RemoveContainer" containerID="727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"
Sep 15 21:18:49.970709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970594    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2pbnlssha5v31\""
Sep 15 21:18:49.970709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970731    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:18:49.970980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:18:49.971044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970968    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a"} err="failed to get container status \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": rpc error: code = NotFound desc = could not find container \"727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a\": container with ID starting with 727ca57b4b19e8fca0a72085e673e218607a93b864d4521b10f5e4c872ecad0a not found: ID does not exist"
Sep 15 21:18:49.971044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.970994    2578 scope.go:117] "RemoveContainer" containerID="90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"
Sep 15 21:18:49.971320 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:18:49.971320 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971263    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:18:49.971454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:18:49.971509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971456    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45"} err="failed to get container status \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": rpc error: code = NotFound desc = could not find container \"90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45\": container with ID starting with 90599d398de242ac19b260a599c8b59bcb9d04c130a4eeec6879324615270a45 not found: ID does not exist"
Sep 15 21:18:49.971509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971480    2578 scope.go:117] "RemoveContainer" containerID="0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"
Sep 15 21:18:49.971617 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971576    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:18:49.971975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.971949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057"} err="failed to get container status \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": rpc error: code = NotFound desc = could not find container \"0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057\": container with ID starting with 0cbef4609035e483731edaaeb6b349304cedbc08d3d0757be2dfae7ad346e057 not found: ID does not exist"
Sep 15 21:18:49.972264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.972246    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:18:49.973941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.973195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:18:49.976216 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.976194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:18:49.979602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:49.979579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:50.050603 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050603 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050971 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050971 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050971 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.050971 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.050971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.051277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.051202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4z9\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-kube-api-access-rf4z9\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.075748 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.075699    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d86865-26b7-44ca-83ba-f537225c8645" path="/var/lib/kubelet/pods/a9d86865-26b7-44ca-83ba-f537225c8645/volumes"
Sep 15 21:18:50.152121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152276    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4z9\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-kube-api-access-rf4z9\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.152758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.153245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.152999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.153245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.153021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.154031 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.154006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.154898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.154296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.156141 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.155755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.156251 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.156136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.156251 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.156192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.156651 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.156612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.156788 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.156668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.157841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.157796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.158051 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.158005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.158684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.158641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.158929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.158889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.159224 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.159201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.159621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.159591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.160459 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.160436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.160567 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.160527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.160699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.160678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.161240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.161205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.163206 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.163179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4z9\" (UniqueName: \"kubernetes.io/projected/b95ea1dd-7a20-47c6-b614-72bb6fd3e09a-kube-api-access-rf4z9\") pod \"prometheus-k8s-0\" (UID: \"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.281980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.281927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:18:50.574213 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.574124    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:18:50.577387 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:18:50.577356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95ea1dd_7a20_47c6_b614_72bb6fd3e09a.slice/crio-c5a52f5ffe1be4fbfe8da690c748df3b99b1f3898cb8ba68e7d52cfef5aec907 WatchSource:0}: Error finding container c5a52f5ffe1be4fbfe8da690c748df3b99b1f3898cb8ba68e7d52cfef5aec907: Status 404 returned error can't find the container with id c5a52f5ffe1be4fbfe8da690c748df3b99b1f3898cb8ba68e7d52cfef5aec907
Sep 15 21:18:50.908509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.908421    2578 generic.go:356] "Generic (PLEG): container finished" podID="b95ea1dd-7a20-47c6-b614-72bb6fd3e09a" containerID="083b3931fb6680cd94b6f0e6b6a6c5235e970462ad603e4a03360a1d0fe8e3e5" exitCode=0
Sep 15 21:18:50.908509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.908487    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerDied","Data":"083b3931fb6680cd94b6f0e6b6a6c5235e970462ad603e4a03360a1d0fe8e3e5"}
Sep 15 21:18:50.909047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:50.908567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"c5a52f5ffe1be4fbfe8da690c748df3b99b1f3898cb8ba68e7d52cfef5aec907"}
Sep 15 21:18:51.916429 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.916382    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8" event={"ID":"cade5465-0375-4250-9dac-31e5503d0453","Type":"ContainerStarted","Data":"ebdfca75f31b942828c3385fbaa41fa1fcce4fa142c34b8c4561ee2ff3fa3d0b"}
Sep 15 21:18:51.916931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.916486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8" event={"ID":"cade5465-0375-4250-9dac-31e5503d0453","Type":"ContainerStarted","Data":"58d344347f54fe0098b22ddca760e57f796e55a7e2994aa995521b98c30374b0"}
Sep 15 21:18:51.916931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.916517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8" event={"ID":"cade5465-0375-4250-9dac-31e5503d0453","Type":"ContainerStarted","Data":"64d56f560c17937cb9d014259c6fb945da6204d097ce52d7d3a617cda28b7f3f"}
Sep 15 21:18:51.919696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"4f4d877924b7bf2ad3aa6635483bce3c865887e4a8382a56271da6110eabf2e0"}
Sep 15 21:18:51.919832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"bf68513a5eae7170109b534d73b3b43560df9f1f0773198dc7f22ae76349498c"}
Sep 15 21:18:51.919832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"aad35410f2cc22d059860151ef01481c12af8d10c22b33ac4e61152df130a5ac"}
Sep 15 21:18:51.919832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"d716427cbdf4a2a7c3d38848d06ceafffbe2a08c647554bcdbab60d63c44f00e"}
Sep 15 21:18:51.919832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"849ebf4cfefa800aba149e61e3d4a747b5b8ed99ea0478459b70b20cc5bdbafe"}
Sep 15 21:18:51.919959 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.919844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b95ea1dd-7a20-47c6-b614-72bb6fd3e09a","Type":"ContainerStarted","Data":"16b98d0fbc08a74a0de8c89dcdb701446bd2ac27ceecf42ed79094a16d5b4d20"}
Sep 15 21:18:51.936857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.936804    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5699449f55-t62h8" podStartSLOduration=1.6881698040000002 podStartE2EDuration="3.936789179s" totalImagesPullingTime="2.248619375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:51.935558201 +0000 UTC m=+160.336531449" watchObservedRunningTime="2026-09-15 21:18:51.936789179 +0000 UTC m=+160.337762425"
Sep 15 21:18:51.965731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:51.965283    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.944153584 podStartE2EDuration="2.965259368s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.105784ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:18:51.963436459 +0000 UTC m=+160.364409707" watchObservedRunningTime="2026-09-15 21:18:51.965259368 +0000 UTC m=+160.366232616"
Sep 15 21:18:55.282476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:18:55.282427    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:19:50.282430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:19:50.282387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:19:50.297939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:19:50.297897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:19:51.187593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:19:51.187514    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:21:11.963393 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:21:11.963366    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:22:00.201117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.201083    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tjdrr"]
Sep 15 21:22:00.204287 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.204271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.206029 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.206010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 21:22:00.206141 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.206043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 21:22:00.206253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.206238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m8vlw\""
Sep 15 21:22:00.214689 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.214656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tjdrr"]
Sep 15 21:22:00.292522 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.292488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstl8\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-kube-api-access-jstl8\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.292690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.292583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-bound-sa-token\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.393780 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.393705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstl8\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-kube-api-access-jstl8\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.393972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.393817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-bound-sa-token\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.402293 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.402260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-bound-sa-token\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.402435 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.402383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstl8\" (UniqueName: \"kubernetes.io/projected/d70352ce-7ad7-402b-9fcb-8de97e73a398-kube-api-access-jstl8\") pod \"cert-manager-55b4d954d5-tjdrr\" (UID: \"d70352ce-7ad7-402b-9fcb-8de97e73a398\") " pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.533530 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.533435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tjdrr"
Sep 15 21:22:00.807389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.807214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tjdrr"]
Sep 15 21:22:00.810057 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:22:00.810024    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70352ce_7ad7_402b_9fcb_8de97e73a398.slice/crio-c22a397e067a2e5b59dda13fa78aff0e6f7a9063f202fcb50d1916fc5d1c0584 WatchSource:0}: Error finding container c22a397e067a2e5b59dda13fa78aff0e6f7a9063f202fcb50d1916fc5d1c0584: Status 404 returned error can't find the container with id c22a397e067a2e5b59dda13fa78aff0e6f7a9063f202fcb50d1916fc5d1c0584
Sep 15 21:22:00.812042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:00.812025    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:22:01.705976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:01.705930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tjdrr" event={"ID":"d70352ce-7ad7-402b-9fcb-8de97e73a398","Type":"ContainerStarted","Data":"c22a397e067a2e5b59dda13fa78aff0e6f7a9063f202fcb50d1916fc5d1c0584"}
Sep 15 21:22:04.720542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:04.720460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tjdrr" event={"ID":"d70352ce-7ad7-402b-9fcb-8de97e73a398","Type":"ContainerStarted","Data":"145a9bee89d2805d0c57af31e4f46351dfe6202d45f965cf67b271d6cdc2d0d3"}
Sep 15 21:22:04.739738 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:04.738162    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tjdrr" podStartSLOduration=1.8773481699999999 podStartE2EDuration="4.738140984s" totalImagesPullingTime="2.860792814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:04.735929151 +0000 UTC m=+353.136902398" watchObservedRunningTime="2026-09-15 21:22:04.738140984 +0000 UTC m=+353.139114231"
Sep 15 21:22:23.334430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.334393    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"]
Sep 15 21:22:23.344865 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.344829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.349795 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.349761    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:22:23.349945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.349926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p85r2\""
Sep 15 21:22:23.350007 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.349995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:22:23.350064 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.350023    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:22:23.350120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.350042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:22:23.366356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.366304    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"]
Sep 15 21:22:23.498815 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.498770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.499007 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.498844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.499007 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.498946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.499007 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.498981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5phh\" (UniqueName: \"kubernetes.io/projected/27835161-a6a8-4f98-bc79-8825748e6b66-kube-api-access-x5phh\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.499120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.499007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.599969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.600008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5phh\" (UniqueName: \"kubernetes.io/projected/27835161-a6a8-4f98-bc79-8825748e6b66-kube-api-access-x5phh\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.600029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.600056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600392 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.600112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.600473 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.600446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-manifests\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.602229 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.602201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27835161-a6a8-4f98-bc79-8825748e6b66-tmp\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.602481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.602463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-webhook-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.602551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.602532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27835161-a6a8-4f98-bc79-8825748e6b66-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.607862 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.607842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5phh\" (UniqueName: \"kubernetes.io/projected/27835161-a6a8-4f98-bc79-8825748e6b66-kube-api-access-x5phh\") pod \"opendatahub-operator-controller-manager-596bc79d9b-f6zdk\" (UID: \"27835161-a6a8-4f98-bc79-8825748e6b66\") " pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.655321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.655294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:23.939396 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:23.939373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"]
Sep 15 21:22:23.942180 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:22:23.942153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27835161_a6a8_4f98_bc79_8825748e6b66.slice/crio-72022d5071788f8cd10f9df10ea6f2d2cd32a230702eff939ec2595dcbd537d8 WatchSource:0}: Error finding container 72022d5071788f8cd10f9df10ea6f2d2cd32a230702eff939ec2595dcbd537d8: Status 404 returned error can't find the container with id 72022d5071788f8cd10f9df10ea6f2d2cd32a230702eff939ec2595dcbd537d8
Sep 15 21:22:24.808575 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:24.808538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk" event={"ID":"27835161-a6a8-4f98-bc79-8825748e6b66","Type":"ContainerStarted","Data":"72022d5071788f8cd10f9df10ea6f2d2cd32a230702eff939ec2595dcbd537d8"}
Sep 15 21:22:27.824117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:27.824081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk" event={"ID":"27835161-a6a8-4f98-bc79-8825748e6b66","Type":"ContainerStarted","Data":"8cb23c356d5b58114bcb4414b6dd4ed8120995ce1c4b93ee1102c3029052ff30"}
Sep 15 21:22:28.829781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:28.829747    2578 generic.go:356] "Generic (PLEG): container finished" podID="27835161-a6a8-4f98-bc79-8825748e6b66" containerID="8cb23c356d5b58114bcb4414b6dd4ed8120995ce1c4b93ee1102c3029052ff30" exitCode=0
Sep 15 21:22:28.830169 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:28.829796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk" event={"ID":"27835161-a6a8-4f98-bc79-8825748e6b66","Type":"ContainerDied","Data":"8cb23c356d5b58114bcb4414b6dd4ed8120995ce1c4b93ee1102c3029052ff30"}
Sep 15 21:22:29.836237 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:29.836199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk" event={"ID":"27835161-a6a8-4f98-bc79-8825748e6b66","Type":"ContainerStarted","Data":"768b181d2627c5dfa6ca43a5ed6f6177d958b0c80ceb660323862083de73fea3"}
Sep 15 21:22:29.836710 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:29.836284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:29.862192 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:29.862140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk" podStartSLOduration=2.3891314599999998 podStartE2EDuration="6.86212422s" totalImagesPullingTime="3.45427088s" totalInitContainerRuntime="1.01872188s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:29.860630796 +0000 UTC m=+378.261604042" watchObservedRunningTime="2026-09-15 21:22:29.86212422 +0000 UTC m=+378.263097467"
Sep 15 21:22:40.844131 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:40.844095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-596bc79d9b-f6zdk"
Sep 15 21:22:42.536240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.536206    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"]
Sep 15 21:22:42.538888 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.538866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.543743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.543705    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 21:22:42.544704 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.544679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:22:42.544865 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.544740    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lbk2h\""
Sep 15 21:22:42.545236 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.545216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 21:22:42.545312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.545243    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 21:22:42.547394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.547369    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:22:42.552623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.552599    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"]
Sep 15 21:22:42.554859 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.554835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-metrics-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.554957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.554897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r849c\" (UniqueName: \"kubernetes.io/projected/d66337a6-479c-4a17-bf61-960409c179de-kube-api-access-r849c\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.555003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.554976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.555054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.555037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d66337a6-479c-4a17-bf61-960409c179de-manager-config\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.655763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.655712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r849c\" (UniqueName: \"kubernetes.io/projected/d66337a6-479c-4a17-bf61-960409c179de-kube-api-access-r849c\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.655946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.655775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.655946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.655804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d66337a6-479c-4a17-bf61-960409c179de-manager-config\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.655946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.655841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-metrics-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.656593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.656567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d66337a6-479c-4a17-bf61-960409c179de-manager-config\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.658245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.658216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-metrics-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.658353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.658293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d66337a6-479c-4a17-bf61-960409c179de-cert\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.685119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.685085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r849c\" (UniqueName: \"kubernetes.io/projected/d66337a6-479c-4a17-bf61-960409c179de-kube-api-access-r849c\") pod \"lws-controller-manager-dccb565dd-96nhw\" (UID: \"d66337a6-479c-4a17-bf61-960409c179de\") " pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:42.848118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:42.848034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:43.025836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.025805    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"]
Sep 15 21:22:43.028207 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.028191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.030094 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.030075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x584x\""
Sep 15 21:22:43.030405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.030386    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:22:43.030825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.030809    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:22:43.030931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.030810    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:22:43.031016 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.031004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:22:43.046115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.046091    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"]
Sep 15 21:22:43.058344 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.058314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tmp\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.058500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.058361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.058500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.058423    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2mz\" (UniqueName: \"kubernetes.io/projected/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-kube-api-access-xc2mz\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.136388 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.136311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"]
Sep 15 21:22:43.139682 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:22:43.139658    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66337a6_479c_4a17_bf61_960409c179de.slice/crio-73c9b2834052c5db2b288fc862ee3383a1298dcafc283e66870f40a717450046 WatchSource:0}: Error finding container 73c9b2834052c5db2b288fc862ee3383a1298dcafc283e66870f40a717450046: Status 404 returned error can't find the container with id 73c9b2834052c5db2b288fc862ee3383a1298dcafc283e66870f40a717450046
Sep 15 21:22:43.159786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.159756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.159950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.159802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2mz\" (UniqueName: \"kubernetes.io/projected/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-kube-api-access-xc2mz\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.159950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.159878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tmp\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.162062 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.162038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tmp\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.162215 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.162195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-tls-certs\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.169852 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.169828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2mz\" (UniqueName: \"kubernetes.io/projected/23a497b4-8e8b-4bf3-a03f-6af1bdd58259-kube-api-access-xc2mz\") pod \"kube-auth-proxy-58bcc99dc6-z4mdg\" (UID: \"23a497b4-8e8b-4bf3-a03f-6af1bdd58259\") " pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.337743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.337701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"
Sep 15 21:22:43.609657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.609611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg"]
Sep 15 21:22:43.616756 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:22:43.616709    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a497b4_8e8b_4bf3_a03f_6af1bdd58259.slice/crio-6360727d6e2d28abdff996d58a597bf784e42e647fc3b7a24596b678980b0230 WatchSource:0}: Error finding container 6360727d6e2d28abdff996d58a597bf784e42e647fc3b7a24596b678980b0230: Status 404 returned error can't find the container with id 6360727d6e2d28abdff996d58a597bf784e42e647fc3b7a24596b678980b0230
Sep 15 21:22:43.905439 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.905303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg" event={"ID":"23a497b4-8e8b-4bf3-a03f-6af1bdd58259","Type":"ContainerStarted","Data":"6360727d6e2d28abdff996d58a597bf784e42e647fc3b7a24596b678980b0230"}
Sep 15 21:22:43.907060 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:43.906997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw" event={"ID":"d66337a6-479c-4a17-bf61-960409c179de","Type":"ContainerStarted","Data":"73c9b2834052c5db2b288fc862ee3383a1298dcafc283e66870f40a717450046"}
Sep 15 21:22:47.928444 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:47.928405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg" event={"ID":"23a497b4-8e8b-4bf3-a03f-6af1bdd58259","Type":"ContainerStarted","Data":"a66a47e3c0365194cb915477b053c60f4d40743627e7ebf48948ed53c1864c59"}
Sep 15 21:22:47.929918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:47.929890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw" event={"ID":"d66337a6-479c-4a17-bf61-960409c179de","Type":"ContainerStarted","Data":"22f789bf87172c9faf0de3baa4df4a42422028c8d458f5c79fd1961282dca07b"}
Sep 15 21:22:47.930045 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:47.930026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:22:47.946156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:47.946094    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58bcc99dc6-z4mdg" podStartSLOduration=2.401143248 podStartE2EDuration="5.946075549s" totalImagesPullingTime="3.544932301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:47.944578334 +0000 UTC m=+396.345551580" watchObservedRunningTime="2026-09-15 21:22:47.946075549 +0000 UTC m=+396.347048798"
Sep 15 21:22:47.962104 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:47.962049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw" podStartSLOduration=1.9439922950000001 podStartE2EDuration="5.962028905s" totalImagesPullingTime="4.01803661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:47.961365448 +0000 UTC m=+396.362338693" watchObservedRunningTime="2026-09-15 21:22:47.962028905 +0000 UTC m=+396.363002151"
Sep 15 21:22:58.935953 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:22:58.935918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-dccb565dd-96nhw"
Sep 15 21:24:26.938192 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.938155    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"]
Sep 15 21:24:26.940600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.940579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:26.942908 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.942884    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:24:26.943355 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.943331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 21:24:26.943454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.943355    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q9527\""
Sep 15 21:24:26.943454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.943409    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:24:26.943564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.943532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 21:24:26.952263 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:26.952240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"]
Sep 15 21:24:27.018047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.018011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13de072d-f42d-46e1-a663-02fbd6939e8c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.018206 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.018068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.018206 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.018092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms75m\" (UniqueName: \"kubernetes.io/projected/13de072d-f42d-46e1-a663-02fbd6939e8c-kube-api-access-ms75m\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.119510 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.119477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.119680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.119516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms75m\" (UniqueName: \"kubernetes.io/projected/13de072d-f42d-46e1-a663-02fbd6939e8c-kube-api-access-ms75m\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.119680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.119584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13de072d-f42d-46e1-a663-02fbd6939e8c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.119680 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:24:27.119631    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 21:24:27.119852 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:24:27.119707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert podName:13de072d-f42d-46e1-a663-02fbd6939e8c nodeName:}" failed. No retries permitted until 2026-09-15 21:24:27.619689148 +0000 UTC m=+496.020662372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-wnrnd" (UID: "13de072d-f42d-46e1-a663-02fbd6939e8c") : secret "plugin-serving-cert" not found
Sep 15 21:24:27.120166 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.120148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13de072d-f42d-46e1-a663-02fbd6939e8c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.130447 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.130422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms75m\" (UniqueName: \"kubernetes.io/projected/13de072d-f42d-46e1-a663-02fbd6939e8c-kube-api-access-ms75m\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.623997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.623965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.626378 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.626357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13de072d-f42d-46e1-a663-02fbd6939e8c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wnrnd\" (UID: \"13de072d-f42d-46e1-a663-02fbd6939e8c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:27.849883 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:27.849845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"
Sep 15 21:24:28.140209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:28.140177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd"]
Sep 15 21:24:28.145320 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:24:28.145251    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13de072d_f42d_46e1_a663_02fbd6939e8c.slice/crio-80285d6371215554750987d507e51cf89b88b847b11856e5224ad0bd4c7b6710 WatchSource:0}: Error finding container 80285d6371215554750987d507e51cf89b88b847b11856e5224ad0bd4c7b6710: Status 404 returned error can't find the container with id 80285d6371215554750987d507e51cf89b88b847b11856e5224ad0bd4c7b6710
Sep 15 21:24:28.384575 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:28.384542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd" event={"ID":"13de072d-f42d-46e1-a663-02fbd6939e8c","Type":"ContainerStarted","Data":"80285d6371215554750987d507e51cf89b88b847b11856e5224ad0bd4c7b6710"}
Sep 15 21:24:33.414882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:33.414844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd" event={"ID":"13de072d-f42d-46e1-a663-02fbd6939e8c","Type":"ContainerStarted","Data":"e3bf774fd405bc33bec3270f10bd6ede1031175df1c3a83d1ad360e528886939"}
Sep 15 21:24:33.453555 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:24:33.453500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wnrnd" podStartSLOduration=2.750232157 podStartE2EDuration="7.453482699s" totalImagesPullingTime="4.703250542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:24:33.452757841 +0000 UTC m=+501.853731078" watchObservedRunningTime="2026-09-15 21:24:33.453482699 +0000 UTC m=+501.854455944"
Sep 15 21:25:24.335509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.335473    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:24.338711 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.338694    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.340357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.340335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 21:25:24.345851 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.345826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:24.425138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.425102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:24.425611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.425584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.425668 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.425653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5g9g\" (UniqueName: \"kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.526868 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.526824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.527015 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.526881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5g9g\" (UniqueName: \"kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.527469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.527450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.534580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.534547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5g9g\" (UniqueName: \"kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g\") pod \"limitador-limitador-8588749977-k5zx7\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") " pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.649560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.649471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:24.922948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:24.922924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:24.924813 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:25:24.924786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f01464_0f9a_489a_aa7b_1adbe81de9f6.slice/crio-c7471fc256c8cb9d8d55912cd68b0ca550f3f6bc84ac9ab23967a7d339f841da WatchSource:0}: Error finding container c7471fc256c8cb9d8d55912cd68b0ca550f3f6bc84ac9ab23967a7d339f841da: Status 404 returned error can't find the container with id c7471fc256c8cb9d8d55912cd68b0ca550f3f6bc84ac9ab23967a7d339f841da
Sep 15 21:25:25.299351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.299322    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:25.303817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.303795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:25.305615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.305597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bdwpx\""
Sep 15 21:25:25.307829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.307804    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:25.434544 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.434502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zps8\" (UniqueName: \"kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8\") pod \"authorino-59c974bd48-2zlsg\" (UID: \"123903a3-b258-4b97-9c70-7b7a5a406ad7\") " pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:25.535327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.535282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zps8\" (UniqueName: \"kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8\") pod \"authorino-59c974bd48-2zlsg\" (UID: \"123903a3-b258-4b97-9c70-7b7a5a406ad7\") " pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:25.543402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.543369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zps8\" (UniqueName: \"kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8\") pod \"authorino-59c974bd48-2zlsg\" (UID: \"123903a3-b258-4b97-9c70-7b7a5a406ad7\") " pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:25.613825 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.613737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:25.662281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.662239    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" event={"ID":"37f01464-0f9a-489a-aa7b-1adbe81de9f6","Type":"ContainerStarted","Data":"c7471fc256c8cb9d8d55912cd68b0ca550f3f6bc84ac9ab23967a7d339f841da"}
Sep 15 21:25:25.882680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:25.882606    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:25.885847 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:25:25.885818    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod123903a3_b258_4b97_9c70_7b7a5a406ad7.slice/crio-7477e5e7778ba2a020b21612ed10ba1f5c2cb1556535b0f988436184bea0d78d WatchSource:0}: Error finding container 7477e5e7778ba2a020b21612ed10ba1f5c2cb1556535b0f988436184bea0d78d: Status 404 returned error can't find the container with id 7477e5e7778ba2a020b21612ed10ba1f5c2cb1556535b0f988436184bea0d78d
Sep 15 21:25:26.670270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:26.670207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2zlsg" event={"ID":"123903a3-b258-4b97-9c70-7b7a5a406ad7","Type":"ContainerStarted","Data":"7477e5e7778ba2a020b21612ed10ba1f5c2cb1556535b0f988436184bea0d78d"}
Sep 15 21:25:28.681976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:28.681895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2zlsg" event={"ID":"123903a3-b258-4b97-9c70-7b7a5a406ad7","Type":"ContainerStarted","Data":"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"}
Sep 15 21:25:28.683468 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:28.683444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" event={"ID":"37f01464-0f9a-489a-aa7b-1adbe81de9f6","Type":"ContainerStarted","Data":"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"}
Sep 15 21:25:28.683607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:28.683586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:28.700063 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:28.699995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2zlsg" podStartSLOduration=1.295889056 podStartE2EDuration="3.699979665s" totalImagesPullingTime="2.404090609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:28.699350147 +0000 UTC m=+557.100323393" watchObservedRunningTime="2026-09-15 21:25:28.699979665 +0000 UTC m=+557.100952911"
Sep 15 21:25:28.721599 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:28.721541    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" podStartSLOduration=1.36386641 podStartE2EDuration="4.721518377s" totalImagesPullingTime="3.357651967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:28.720162951 +0000 UTC m=+557.121136198" watchObservedRunningTime="2026-09-15 21:25:28.721518377 +0000 UTC m=+557.122491627"
Sep 15 21:25:38.580129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:38.580094    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:38.580655 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:38.580281    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" podUID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" containerName="limitador" containerID="cri-o://084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c" gracePeriod=30
Sep 15 21:25:38.581054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:38.580891    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:39.121537 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.121512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:39.153515 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.153436    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file\") pod \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") "
Sep 15 21:25:39.153515 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.153492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5g9g\" (UniqueName: \"kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g\") pod \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\" (UID: \"37f01464-0f9a-489a-aa7b-1adbe81de9f6\") "
Sep 15 21:25:39.153765 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.153735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file" (OuterVolumeSpecName: "config-file") pod "37f01464-0f9a-489a-aa7b-1adbe81de9f6" (UID: "37f01464-0f9a-489a-aa7b-1adbe81de9f6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:25:39.153844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.153826    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37f01464-0f9a-489a-aa7b-1adbe81de9f6-config-file\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:39.155621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.155588    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g" (OuterVolumeSpecName: "kube-api-access-r5g9g") pod "37f01464-0f9a-489a-aa7b-1adbe81de9f6" (UID: "37f01464-0f9a-489a-aa7b-1adbe81de9f6"). InnerVolumeSpecName "kube-api-access-r5g9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:39.254908 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.254869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5g9g\" (UniqueName: \"kubernetes.io/projected/37f01464-0f9a-489a-aa7b-1adbe81de9f6-kube-api-access-r5g9g\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:39.737412 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.737381    2578 generic.go:356] "Generic (PLEG): container finished" podID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" containerID="084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c" exitCode=0
Sep 15 21:25:39.737842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.737428    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7"
Sep 15 21:25:39.737842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.737462    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" event={"ID":"37f01464-0f9a-489a-aa7b-1adbe81de9f6","Type":"ContainerDied","Data":"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"}
Sep 15 21:25:39.737842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.737524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-k5zx7" event={"ID":"37f01464-0f9a-489a-aa7b-1adbe81de9f6","Type":"ContainerDied","Data":"c7471fc256c8cb9d8d55912cd68b0ca550f3f6bc84ac9ab23967a7d339f841da"}
Sep 15 21:25:39.737842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.737552    2578 scope.go:117] "RemoveContainer" containerID="084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"
Sep 15 21:25:39.746342 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.746326    2578 scope.go:117] "RemoveContainer" containerID="084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"
Sep 15 21:25:39.746631 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:25:39.746608    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c\": container with ID starting with 084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c not found: ID does not exist" containerID="084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"
Sep 15 21:25:39.746710 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.746644    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c"} err="failed to get container status \"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c\": rpc error: code = NotFound desc = could not find container \"084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c\": container with ID starting with 084c49978c27f5287bbf9b09d06126978e5de69ffa00e6c1246ba06b9a67a86c not found: ID does not exist"
Sep 15 21:25:39.756997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.756977    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:39.762987 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:39.762963    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-k5zx7"]
Sep 15 21:25:40.075624 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:40.075548    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" path="/var/lib/kubelet/pods/37f01464-0f9a-489a-aa7b-1adbe81de9f6/volumes"
Sep 15 21:25:55.045243 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.045203    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-bzl9v"]
Sep 15 21:25:55.045857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.045773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" containerName="limitador"
Sep 15 21:25:55.045857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.045794    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" containerName="limitador"
Sep 15 21:25:55.045985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.045884    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f01464-0f9a-489a-aa7b-1adbe81de9f6" containerName="limitador"
Sep 15 21:25:55.050270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.050245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.055639 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.055605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-bzl9v"]
Sep 15 21:25:55.139341 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.139303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-bzl9v"]
Sep 15 21:25:55.139586 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:25:55.139568    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-b6jk7], unattached volumes=[], failed to process volumes=[kube-api-access-b6jk7]: context canceled" pod="kuadrant-system/authorino-557f876f69-bzl9v" podUID="64fc5e1b-8e32-4037-a477-4c0061ba3390"
Sep 15 21:25:55.173491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.173460    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6nlz5"]
Sep 15 21:25:55.175758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.175741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.177940 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.177918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 21:25:55.185351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.185328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6nlz5"]
Sep 15 21:25:55.197821 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.197796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jk7\" (UniqueName: \"kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7\") pod \"authorino-557f876f69-bzl9v\" (UID: \"64fc5e1b-8e32-4037-a477-4c0061ba3390\") " pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.282519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.282477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6nlz5"]
Sep 15 21:25:55.282847 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:25:55.282825    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-npxsh tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-6nlz5" podUID="ef0a1d23-51f7-4525-be5f-3854c10fa200"
Sep 15 21:25:55.298328 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.298254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jk7\" (UniqueName: \"kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7\") pod \"authorino-557f876f69-bzl9v\" (UID: \"64fc5e1b-8e32-4037-a477-4c0061ba3390\") " pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.298328 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.298310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxsh\" (UniqueName: \"kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.298502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.298341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.310985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.310953    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jk7\" (UniqueName: \"kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7\") pod \"authorino-557f876f69-bzl9v\" (UID: \"64fc5e1b-8e32-4037-a477-4c0061ba3390\") " pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.315277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.315249    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:25:55.317604 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.317581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.327998 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.327976    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:25:55.399365 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.399325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxsh\" (UniqueName: \"kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.399552 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.399374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nn8\" (UniqueName: \"kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.399552 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.399409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.399552 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.399459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.401910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.401891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.407393 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.407374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxsh\" (UniqueName: \"kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh\") pod \"authorino-6d4dfc649f-6nlz5\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") " pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.500384 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.500346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.500560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.500425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nn8\" (UniqueName: \"kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.502802 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.502780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.508117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.508099    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nn8\" (UniqueName: \"kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8\") pod \"authorino-747c668c6f-s2brw\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") " pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.627002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.626922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:25:55.812323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.812291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.812540 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.812291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.818740 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.818706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:55.821509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.821487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:55.895817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.895742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:25:55.898653 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:25:55.898626    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8a0eeac_3b26_4f30_8559_efb9fad0e583.slice/crio-37f22c7984b1ef4e40a51cc012c80d207705bd928bcb95e3856f7b5cb140d479 WatchSource:0}: Error finding container 37f22c7984b1ef4e40a51cc012c80d207705bd928bcb95e3856f7b5cb140d479: Status 404 returned error can't find the container with id 37f22c7984b1ef4e40a51cc012c80d207705bd928bcb95e3856f7b5cb140d479
Sep 15 21:25:55.904512 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.904490    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6jk7\" (UniqueName: \"kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7\") pod \"64fc5e1b-8e32-4037-a477-4c0061ba3390\" (UID: \"64fc5e1b-8e32-4037-a477-4c0061ba3390\") "
Sep 15 21:25:55.904646 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.904529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npxsh\" (UniqueName: \"kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh\") pod \"ef0a1d23-51f7-4525-be5f-3854c10fa200\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") "
Sep 15 21:25:55.904692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.904664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert\") pod \"ef0a1d23-51f7-4525-be5f-3854c10fa200\" (UID: \"ef0a1d23-51f7-4525-be5f-3854c10fa200\") "
Sep 15 21:25:55.906569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.906544    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7" (OuterVolumeSpecName: "kube-api-access-b6jk7") pod "64fc5e1b-8e32-4037-a477-4c0061ba3390" (UID: "64fc5e1b-8e32-4037-a477-4c0061ba3390"). InnerVolumeSpecName "kube-api-access-b6jk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:55.906740 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.906706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ef0a1d23-51f7-4525-be5f-3854c10fa200" (UID: "ef0a1d23-51f7-4525-be5f-3854c10fa200"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:25:55.906802 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:55.906737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh" (OuterVolumeSpecName: "kube-api-access-npxsh") pod "ef0a1d23-51f7-4525-be5f-3854c10fa200" (UID: "ef0a1d23-51f7-4525-be5f-3854c10fa200"). InnerVolumeSpecName "kube-api-access-npxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:56.006278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.006248    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef0a1d23-51f7-4525-be5f-3854c10fa200-tls-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:56.006278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.006274    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jk7\" (UniqueName: \"kubernetes.io/projected/64fc5e1b-8e32-4037-a477-4c0061ba3390-kube-api-access-b6jk7\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:56.006278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.006284    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxsh\" (UniqueName: \"kubernetes.io/projected/ef0a1d23-51f7-4525-be5f-3854c10fa200-kube-api-access-npxsh\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:56.819118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.819078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747c668c6f-s2brw" event={"ID":"f8a0eeac-3b26-4f30-8559-efb9fad0e583","Type":"ContainerStarted","Data":"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"}
Sep 15 21:25:56.819118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.819105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-bzl9v"
Sep 15 21:25:56.819118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.819118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-6nlz5"
Sep 15 21:25:56.819634 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.819123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747c668c6f-s2brw" event={"ID":"f8a0eeac-3b26-4f30-8559-efb9fad0e583","Type":"ContainerStarted","Data":"37f22c7984b1ef4e40a51cc012c80d207705bd928bcb95e3856f7b5cb140d479"}
Sep 15 21:25:56.837919 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.837852    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-747c668c6f-s2brw" podStartSLOduration=1.334108401 podStartE2EDuration="1.837819906s" totalImagesPullingTime="503.711505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:25:56.835833247 +0000 UTC m=+585.236806494" watchObservedRunningTime="2026-09-15 21:25:56.837819906 +0000 UTC m=+585.238793151"
Sep 15 21:25:56.859787 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.859756    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-bzl9v"]
Sep 15 21:25:56.864644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.864567    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:56.864858 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.864798    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2zlsg" podUID="123903a3-b258-4b97-9c70-7b7a5a406ad7" containerName="authorino" containerID="cri-o://6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5" gracePeriod=30
Sep 15 21:25:56.866732 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.866681    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-bzl9v"]
Sep 15 21:25:56.893875 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.893847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6nlz5"]
Sep 15 21:25:56.897384 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:56.897357    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-6nlz5"]
Sep 15 21:25:57.089194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.089167    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:57.224878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.224838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zps8\" (UniqueName: \"kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8\") pod \"123903a3-b258-4b97-9c70-7b7a5a406ad7\" (UID: \"123903a3-b258-4b97-9c70-7b7a5a406ad7\") "
Sep 15 21:25:57.226929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.226904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8" (OuterVolumeSpecName: "kube-api-access-6zps8") pod "123903a3-b258-4b97-9c70-7b7a5a406ad7" (UID: "123903a3-b258-4b97-9c70-7b7a5a406ad7"). InnerVolumeSpecName "kube-api-access-6zps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:25:57.325881 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.325853    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zps8\" (UniqueName: \"kubernetes.io/projected/123903a3-b258-4b97-9c70-7b7a5a406ad7-kube-api-access-6zps8\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:25:57.825276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.825242    2578 generic.go:356] "Generic (PLEG): container finished" podID="123903a3-b258-4b97-9c70-7b7a5a406ad7" containerID="6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5" exitCode=0
Sep 15 21:25:57.825685 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.825283    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2zlsg"
Sep 15 21:25:57.825685 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.825329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2zlsg" event={"ID":"123903a3-b258-4b97-9c70-7b7a5a406ad7","Type":"ContainerDied","Data":"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"}
Sep 15 21:25:57.825685 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.825369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2zlsg" event={"ID":"123903a3-b258-4b97-9c70-7b7a5a406ad7","Type":"ContainerDied","Data":"7477e5e7778ba2a020b21612ed10ba1f5c2cb1556535b0f988436184bea0d78d"}
Sep 15 21:25:57.825685 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.825386    2578 scope.go:117] "RemoveContainer" containerID="6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"
Sep 15 21:25:57.837672 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.837655    2578 scope.go:117] "RemoveContainer" containerID="6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"
Sep 15 21:25:57.837977 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:25:57.837956    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5\": container with ID starting with 6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5 not found: ID does not exist" containerID="6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"
Sep 15 21:25:57.838037 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.837986    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5"} err="failed to get container status \"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5\": rpc error: code = NotFound desc = could not find container \"6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5\": container with ID starting with 6728893c4310f4e18958da0e51a6627adb5ae25e036d629598592954e4fe45b5 not found: ID does not exist"
Sep 15 21:25:57.848375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.848351    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:57.851734 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:57.851694    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2zlsg"]
Sep 15 21:25:58.075690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:58.075599    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123903a3-b258-4b97-9c70-7b7a5a406ad7" path="/var/lib/kubelet/pods/123903a3-b258-4b97-9c70-7b7a5a406ad7/volumes"
Sep 15 21:25:58.076038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:58.076017    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fc5e1b-8e32-4037-a477-4c0061ba3390" path="/var/lib/kubelet/pods/64fc5e1b-8e32-4037-a477-4c0061ba3390/volumes"
Sep 15 21:25:58.076310 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:25:58.076294    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0a1d23-51f7-4525-be5f-3854c10fa200" path="/var/lib/kubelet/pods/ef0a1d23-51f7-4525-be5f-3854c10fa200/volumes"
Sep 15 21:26:01.553396 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.553361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:01.553808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.553736    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123903a3-b258-4b97-9c70-7b7a5a406ad7" containerName="authorino"
Sep 15 21:26:01.553808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.553748    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="123903a3-b258-4b97-9c70-7b7a5a406ad7" containerName="authorino"
Sep 15 21:26:01.553808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.553809    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="123903a3-b258-4b97-9c70-7b7a5a406ad7" containerName="authorino"
Sep 15 21:26:01.556542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.556524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.558349 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.558321    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:26:01.558498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.558382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:26:01.558498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.558414    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vsdzf\""
Sep 15 21:26:01.558498 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.558479    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:26:01.568049 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.568024    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:01.667633 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.667595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.667867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.667710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs4t6\" (UniqueName: \"kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.667867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.667841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.742916 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.740184    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:26:01.744312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.744280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.751699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.751669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:26:01.769244 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.769210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs4t6\" (UniqueName: \"kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.769405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.769272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.769405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.769304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.771650 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.771622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.771770 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.771660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.778315 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.778291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs4t6\" (UniqueName: \"kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6\") pod \"maas-controller-66897d47ff-w7tm6\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") " pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.867326 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.867240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:01.870338 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.870315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l858c\" (UniqueName: \"kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.870462 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.870350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.870462 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.870373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.971290 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.971252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.971481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.971298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.971481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.971369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l858c\" (UniqueName: \"kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.973738 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.973702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.973860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.973758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:01.978862 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:01.978835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l858c\" (UniqueName: \"kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c\") pod \"maas-controller-bd4b7887f-qskxh\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") " pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:02.055987 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:02.055947    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:02.139052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:02.139021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:02.141295 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:26:02.141271    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28451402_fa46_4819_b43a_a2d36d8902e5.slice/crio-269d42aa18cf2cf188eb8b85b7f07a93300cbe3af35f3435b07306729b3a3571 WatchSource:0}: Error finding container 269d42aa18cf2cf188eb8b85b7f07a93300cbe3af35f3435b07306729b3a3571: Status 404 returned error can't find the container with id 269d42aa18cf2cf188eb8b85b7f07a93300cbe3af35f3435b07306729b3a3571
Sep 15 21:26:02.334132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:02.334100    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:26:02.337598 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:26:02.337570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276a075a_a2bf_47c3_bc81_d1375f92993e.slice/crio-c477000d15120f08ab39035caf039b1b33b6f7bf813fd7dc34c480342be6b606 WatchSource:0}: Error finding container c477000d15120f08ab39035caf039b1b33b6f7bf813fd7dc34c480342be6b606: Status 404 returned error can't find the container with id c477000d15120f08ab39035caf039b1b33b6f7bf813fd7dc34c480342be6b606
Sep 15 21:26:02.849871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:02.849835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qskxh" event={"ID":"276a075a-a2bf-47c3-bc81-d1375f92993e","Type":"ContainerStarted","Data":"c477000d15120f08ab39035caf039b1b33b6f7bf813fd7dc34c480342be6b606"}
Sep 15 21:26:02.851052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:02.851028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66897d47ff-w7tm6" event={"ID":"28451402-fa46-4819-b43a-a2d36d8902e5","Type":"ContainerStarted","Data":"269d42aa18cf2cf188eb8b85b7f07a93300cbe3af35f3435b07306729b3a3571"}
Sep 15 21:26:05.874926 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.874881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qskxh" event={"ID":"276a075a-a2bf-47c3-bc81-d1375f92993e","Type":"ContainerStarted","Data":"9e5147ddd53f0289014f5b89eae128f13a497f842331c3eb11d97277529c7258"}
Sep 15 21:26:05.875364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.875024    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:05.876505 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.876477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66897d47ff-w7tm6" event={"ID":"28451402-fa46-4819-b43a-a2d36d8902e5","Type":"ContainerStarted","Data":"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"}
Sep 15 21:26:05.876628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.876531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:05.915321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.915262    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd4b7887f-qskxh" podStartSLOduration=2.239432 podStartE2EDuration="4.915246613s" totalImagesPullingTime="2.675814613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:05.914439991 +0000 UTC m=+594.315413237" watchObservedRunningTime="2026-09-15 21:26:05.915246613 +0000 UTC m=+594.316219859"
Sep 15 21:26:05.966532 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:05.966477    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66897d47ff-w7tm6" podStartSLOduration=2.0974494 podStartE2EDuration="4.966462381s" totalImagesPullingTime="2.869012981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:05.965355304 +0000 UTC m=+594.366328550" watchObservedRunningTime="2026-09-15 21:26:05.966462381 +0000 UTC m=+594.367435626"
Sep 15 21:26:16.883985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:16.883952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:26:16.884370 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:16.884036    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:16.950159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:16.950121    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:16.950370 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:16.950319    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-66897d47ff-w7tm6" podUID="28451402-fa46-4819-b43a-a2d36d8902e5" containerName="manager" containerID="cri-o://4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745" gracePeriod=30
Sep 15 21:26:17.194580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.194551    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:17.213905 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.213867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zs4t6\" (UniqueName: \"kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6\") pod \"28451402-fa46-4819-b43a-a2d36d8902e5\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") "
Sep 15 21:26:17.214086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.213959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert\") pod \"28451402-fa46-4819-b43a-a2d36d8902e5\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") "
Sep 15 21:26:17.214086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.214023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs\") pod \"28451402-fa46-4819-b43a-a2d36d8902e5\" (UID: \"28451402-fa46-4819-b43a-a2d36d8902e5\") "
Sep 15 21:26:17.216975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.216929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28451402-fa46-4819-b43a-a2d36d8902e5" (UID: "28451402-fa46-4819-b43a-a2d36d8902e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:17.217667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.217625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6" (OuterVolumeSpecName: "kube-api-access-zs4t6") pod "28451402-fa46-4819-b43a-a2d36d8902e5" (UID: "28451402-fa46-4819-b43a-a2d36d8902e5"). InnerVolumeSpecName "kube-api-access-zs4t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:26:17.217667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.217625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "28451402-fa46-4819-b43a-a2d36d8902e5" (UID: "28451402-fa46-4819-b43a-a2d36d8902e5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:26:17.315331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.315300    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-webhook-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:26:17.315331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.315328    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28451402-fa46-4819-b43a-a2d36d8902e5-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:26:17.315331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.315338    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs4t6\" (UniqueName: \"kubernetes.io/projected/28451402-fa46-4819-b43a-a2d36d8902e5-kube-api-access-zs4t6\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:26:17.939913 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.939882    2578 generic.go:356] "Generic (PLEG): container finished" podID="28451402-fa46-4819-b43a-a2d36d8902e5" containerID="4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745" exitCode=0
Sep 15 21:26:17.940319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.939928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66897d47ff-w7tm6"
Sep 15 21:26:17.940319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.939964    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66897d47ff-w7tm6" event={"ID":"28451402-fa46-4819-b43a-a2d36d8902e5","Type":"ContainerDied","Data":"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"}
Sep 15 21:26:17.940319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.940017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66897d47ff-w7tm6" event={"ID":"28451402-fa46-4819-b43a-a2d36d8902e5","Type":"ContainerDied","Data":"269d42aa18cf2cf188eb8b85b7f07a93300cbe3af35f3435b07306729b3a3571"}
Sep 15 21:26:17.940319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.940040    2578 scope.go:117] "RemoveContainer" containerID="4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"
Sep 15 21:26:17.949292 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.949272    2578 scope.go:117] "RemoveContainer" containerID="4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"
Sep 15 21:26:17.949580 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:17.949562    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745\": container with ID starting with 4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745 not found: ID does not exist" containerID="4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"
Sep 15 21:26:17.949626 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.949589    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745"} err="failed to get container status \"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745\": rpc error: code = NotFound desc = could not find container \"4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745\": container with ID starting with 4a33b8a20a13a48b59b91d0606db12596d1acaf495edf2a3b8076d5775e00745 not found: ID does not exist"
Sep 15 21:26:17.961293 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.961264    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:17.963665 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:17.963640    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66897d47ff-w7tm6"]
Sep 15 21:26:18.075205 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:18.075173    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28451402-fa46-4819-b43a-a2d36d8902e5" path="/var/lib/kubelet/pods/28451402-fa46-4819-b43a-a2d36d8902e5/volumes"
Sep 15 21:26:39.542937 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.542853    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:26:39.543383 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.543248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28451402-fa46-4819-b43a-a2d36d8902e5" containerName="manager"
Sep 15 21:26:39.543383 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.543261    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28451402-fa46-4819-b43a-a2d36d8902e5" containerName="manager"
Sep 15 21:26:39.543383 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.543335    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28451402-fa46-4819-b43a-a2d36d8902e5" containerName="manager"
Sep 15 21:26:39.548852 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.548832    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.550640 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.550615    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 21:26:39.550822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.550795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:26:39.550976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.550894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 21:26:39.550976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.550908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sgjsd\""
Sep 15 21:26:39.551662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.551644    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 21:26:39.557591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.557568    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:26:39.618062 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.618029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.618225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.618073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.618225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.618133    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.618225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.618163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27gf\" (UniqueName: \"kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.718929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.718883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.719090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.718949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.719090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.718989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.719090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.719011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27gf\" (UniqueName: \"kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.719090 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:39.719051    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 15 21:26:39.719251 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:39.719111    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 15 21:26:39.719251 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:39.719132    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls podName:b75e843c-1f08-44ab-b565-68517fd7f343 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:40.219112939 +0000 UTC m=+628.620086167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls") pod "maas-api-97c9c8fdc-qm5h6" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343") : secret "maas-api-serving-cert" not found
Sep 15 21:26:39.719251 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:39.719173    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs podName:b75e843c-1f08-44ab-b565-68517fd7f343 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:40.219158308 +0000 UTC m=+628.620131532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs") pod "maas-api-97c9c8fdc-qm5h6" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343") : secret "maas-api-metrics-tls" not found
Sep 15 21:26:39.719659 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.719639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:39.731770 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:39.731743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27gf\" (UniqueName: \"kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:40.223155 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.223114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:40.223346 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.223180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:40.223346 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:40.223278    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 15 21:26:40.223432 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:26:40.223357    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs podName:b75e843c-1f08-44ab-b565-68517fd7f343 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:41.223338497 +0000 UTC m=+629.624311721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs") pod "maas-api-97c9c8fdc-qm5h6" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343") : secret "maas-api-metrics-tls" not found
Sep 15 21:26:40.225601 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.225577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:40.318220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.318187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"]
Sep 15 21:26:40.321557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.321537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.323921 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.323898    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cqkvg\""
Sep 15 21:26:40.324164 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.324149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 21:26:40.337562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.337536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"]
Sep 15 21:26:40.425314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.425271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13951561-032d-4591-88e5-ea2e0befd373-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.425500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.425327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpd9b\" (UniqueName: \"kubernetes.io/projected/13951561-032d-4591-88e5-ea2e0befd373-kube-api-access-vpd9b\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.526978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.526890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13951561-032d-4591-88e5-ea2e0befd373-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.526978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.526940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpd9b\" (UniqueName: \"kubernetes.io/projected/13951561-032d-4591-88e5-ea2e0befd373-kube-api-access-vpd9b\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.527708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.527682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13951561-032d-4591-88e5-ea2e0befd373-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.550123 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.550084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpd9b\" (UniqueName: \"kubernetes.io/projected/13951561-032d-4591-88e5-ea2e0befd373-kube-api-access-vpd9b\") pod \"payload-pre-processing-59fbc47bf8-rfw5s\" (UID: \"13951561-032d-4591-88e5-ea2e0befd373\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.631817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.631774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:40.954589 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:40.954562    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"]
Sep 15 21:26:40.957347 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:26:40.957314    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13951561_032d_4591_88e5_ea2e0befd373.slice/crio-7ae0fc1bc89b66728c29b56897a87192afeed3eb34f43388ce3ff9f8ca70d8ae WatchSource:0}: Error finding container 7ae0fc1bc89b66728c29b56897a87192afeed3eb34f43388ce3ff9f8ca70d8ae: Status 404 returned error can't find the container with id 7ae0fc1bc89b66728c29b56897a87192afeed3eb34f43388ce3ff9f8ca70d8ae
Sep 15 21:26:41.052085 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:41.052046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s" event={"ID":"13951561-032d-4591-88e5-ea2e0befd373","Type":"ContainerStarted","Data":"7ae0fc1bc89b66728c29b56897a87192afeed3eb34f43388ce3ff9f8ca70d8ae"}
Sep 15 21:26:41.234253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:41.234165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:41.236494 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:41.236469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"maas-api-97c9c8fdc-qm5h6\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") " pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:41.360887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:41.360854    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:41.638934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:41.638905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:26:41.641508 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:26:41.641473    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e843c_1f08_44ab_b565_68517fd7f343.slice/crio-71274eac7e4b14eb7d92d47121af020fc59285d404212bdc7571901e6f31682b WatchSource:0}: Error finding container 71274eac7e4b14eb7d92d47121af020fc59285d404212bdc7571901e6f31682b: Status 404 returned error can't find the container with id 71274eac7e4b14eb7d92d47121af020fc59285d404212bdc7571901e6f31682b
Sep 15 21:26:42.059599 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:42.059538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" event={"ID":"b75e843c-1f08-44ab-b565-68517fd7f343","Type":"ContainerStarted","Data":"71274eac7e4b14eb7d92d47121af020fc59285d404212bdc7571901e6f31682b"}
Sep 15 21:26:43.066546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:43.066510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s" event={"ID":"13951561-032d-4591-88e5-ea2e0befd373","Type":"ContainerStarted","Data":"897db7525015a875b993bc1ee5406fd9ce4fede1d226fc9b225a1f409ffdc5db"}
Sep 15 21:26:43.066987 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:43.066638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:26:43.087751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:43.087682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s" podStartSLOduration=1.841409445 podStartE2EDuration="3.087665985s" totalImagesPullingTime="1.24625654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:43.086866294 +0000 UTC m=+631.487839539" watchObservedRunningTime="2026-09-15 21:26:43.087665985 +0000 UTC m=+631.488639228"
Sep 15 21:26:44.075183 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:44.075147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" event={"ID":"b75e843c-1f08-44ab-b565-68517fd7f343","Type":"ContainerStarted","Data":"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"}
Sep 15 21:26:44.094181 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:44.094128    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" podStartSLOduration=3.142462254 podStartE2EDuration="5.094113312s" totalImagesPullingTime="1.951651058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:44.09211258 +0000 UTC m=+632.493085828" watchObservedRunningTime="2026-09-15 21:26:44.094113312 +0000 UTC m=+632.495086603"
Sep 15 21:26:45.078214 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:45.078182    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:51.088581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:51.088496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:26:54.075698 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:26:54.075669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-rfw5s"
Sep 15 21:27:04.013465 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.013424    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"]
Sep 15 21:27:04.016961 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.016940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.019191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.019152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:27:04.019191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.019183    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fxwc6\""
Sep 15 21:27:04.019430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.019232    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:27:04.019430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.019261    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 21:27:04.027063 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.027037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"]
Sep 15 21:27:04.036663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.036803 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw74\" (UniqueName: \"kubernetes.io/projected/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-kube-api-access-7jw74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.036803 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.036890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.036890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.036968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.036889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.137706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.137918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw74\" (UniqueName: \"kubernetes.io/projected/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-kube-api-access-7jw74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.137918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.137918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.138108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.138108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.137998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.138281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.138258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.138364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.138347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.138402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.138376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.139953 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.139934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.140259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.140244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.145760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.145740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw74\" (UniqueName: \"kubernetes.io/projected/0e1e84e3-657d-45d4-a2ab-ef70a38adb48-kube-api-access-7jw74\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld\" (UID: \"0e1e84e3-657d-45d4-a2ab-ef70a38adb48\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.328056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.327972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:04.610062 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.610025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"]
Sep 15 21:27:04.611419 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:27:04.611390    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1e84e3_657d_45d4_a2ab_ef70a38adb48.slice/crio-7f123091f4e6c7a3f16302fc0e92792b3312bd049b50105d0a4b472f37b14260 WatchSource:0}: Error finding container 7f123091f4e6c7a3f16302fc0e92792b3312bd049b50105d0a4b472f37b14260: Status 404 returned error can't find the container with id 7f123091f4e6c7a3f16302fc0e92792b3312bd049b50105d0a4b472f37b14260
Sep 15 21:27:04.612970 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:04.612951    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:27:05.180683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:05.180645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld" event={"ID":"0e1e84e3-657d-45d4-a2ab-ef70a38adb48","Type":"ContainerStarted","Data":"7f123091f4e6c7a3f16302fc0e92792b3312bd049b50105d0a4b472f37b14260"}
Sep 15 21:27:06.194617 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:06.194578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld" event={"ID":"0e1e84e3-657d-45d4-a2ab-ef70a38adb48","Type":"ContainerStarted","Data":"6d1e4ff6c00ae9f3a34d36826c751af25b73db1a35099b9b36677b46158818f8"}
Sep 15 21:27:06.216600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:06.216554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld" podStartSLOduration=2.134236482 podStartE2EDuration="3.21653661s" totalImagesPullingTime="1.082300128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:06.214195606 +0000 UTC m=+654.615168849" watchObservedRunningTime="2026-09-15 21:27:06.21653661 +0000 UTC m=+654.617509856"
Sep 15 21:27:14.328889 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:14.328850    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:14.329314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:14.328919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:14.341486 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:14.341458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:15.254156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:15.254125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld"
Sep 15 21:27:24.512027 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.511991    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"]
Sep 15 21:27:24.519088 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.519064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.520864 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.520839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 21:27:24.527357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.527331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"]
Sep 15 21:27:24.635002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.634965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.635221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.635023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6512e4-09f8-4e73-aef6-0051114c32b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.635221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.635049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pg6v\" (UniqueName: \"kubernetes.io/projected/ba6512e4-09f8-4e73-aef6-0051114c32b3-kube-api-access-2pg6v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.635221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.635106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.635221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.635157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.635429 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.635273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pg6v\" (UniqueName: \"kubernetes.io/projected/ba6512e4-09f8-4e73-aef6-0051114c32b3-kube-api-access-2pg6v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736805 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736805 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736805 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.736805 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.736643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6512e4-09f8-4e73-aef6-0051114c32b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.737023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.737003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.737076 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.737032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.737127 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.737087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.738896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.738874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6512e4-09f8-4e73-aef6-0051114c32b3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.739166 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.739147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6512e4-09f8-4e73-aef6-0051114c32b3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.744187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.744164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pg6v\" (UniqueName: \"kubernetes.io/projected/ba6512e4-09f8-4e73-aef6-0051114c32b3-kube-api-access-2pg6v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj\" (UID: \"ba6512e4-09f8-4e73-aef6-0051114c32b3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:24.830840 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:24.830750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:25.105450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:25.105242    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"]
Sep 15 21:27:25.107921 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:27:25.107888    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6512e4_09f8_4e73_aef6_0051114c32b3.slice/crio-c5f970c11148305d52c2a7b8f4859f1658c3f70b88dd9d9e16090b45146dac4a WatchSource:0}: Error finding container c5f970c11148305d52c2a7b8f4859f1658c3f70b88dd9d9e16090b45146dac4a: Status 404 returned error can't find the container with id c5f970c11148305d52c2a7b8f4859f1658c3f70b88dd9d9e16090b45146dac4a
Sep 15 21:27:25.291284 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:25.291247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj" event={"ID":"ba6512e4-09f8-4e73-aef6-0051114c32b3","Type":"ContainerStarted","Data":"c5f970c11148305d52c2a7b8f4859f1658c3f70b88dd9d9e16090b45146dac4a"}
Sep 15 21:27:26.298845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:26.298797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj" event={"ID":"ba6512e4-09f8-4e73-aef6-0051114c32b3","Type":"ContainerStarted","Data":"58001ecd60678d2cab7868ba661d6fc875bf4fbfc287dde442c54a9b1848cb49"}
Sep 15 21:27:26.317692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:26.317637    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj" podStartSLOduration=2.125161191 podStartE2EDuration="2.317620011s" totalImagesPullingTime="192.45882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:26.316322466 +0000 UTC m=+674.717295712" watchObservedRunningTime="2026-09-15 21:27:26.317620011 +0000 UTC m=+674.718593257"
Sep 15 21:27:34.831802 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:34.831761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:34.832209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:34.831831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:34.844302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:34.844271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:27:35.355784 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:27:35.355752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj"
Sep 15 21:28:55.557690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.557654    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85f847c588-ws6zl"]
Sep 15 21:28:55.561327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.561308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.573595 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.573569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85f847c588-ws6zl"]
Sep 15 21:28:55.629831 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.629790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b80d04e9-56f1-49c7-801c-adfa3daad9f2-tls-cert\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.630024 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.629877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h577m\" (UniqueName: \"kubernetes.io/projected/b80d04e9-56f1-49c7-801c-adfa3daad9f2-kube-api-access-h577m\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.730427 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.730390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b80d04e9-56f1-49c7-801c-adfa3daad9f2-tls-cert\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.730612 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.730460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h577m\" (UniqueName: \"kubernetes.io/projected/b80d04e9-56f1-49c7-801c-adfa3daad9f2-kube-api-access-h577m\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.732878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.732847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b80d04e9-56f1-49c7-801c-adfa3daad9f2-tls-cert\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.738607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.738580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h577m\" (UniqueName: \"kubernetes.io/projected/b80d04e9-56f1-49c7-801c-adfa3daad9f2-kube-api-access-h577m\") pod \"authorino-85f847c588-ws6zl\" (UID: \"b80d04e9-56f1-49c7-801c-adfa3daad9f2\") " pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:55.871080 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:55.871002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85f847c588-ws6zl"
Sep 15 21:28:56.146405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:56.146272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85f847c588-ws6zl"]
Sep 15 21:28:56.148695 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:28:56.148657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80d04e9_56f1_49c7_801c_adfa3daad9f2.slice/crio-da656f3f5d5fb6d8f5bbd94b22bd99b432722256da2bea1c4ef7fc2fe23b8d6a WatchSource:0}: Error finding container da656f3f5d5fb6d8f5bbd94b22bd99b432722256da2bea1c4ef7fc2fe23b8d6a: Status 404 returned error can't find the container with id da656f3f5d5fb6d8f5bbd94b22bd99b432722256da2bea1c4ef7fc2fe23b8d6a
Sep 15 21:28:56.736291 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:56.736257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85f847c588-ws6zl" event={"ID":"b80d04e9-56f1-49c7-801c-adfa3daad9f2","Type":"ContainerStarted","Data":"da656f3f5d5fb6d8f5bbd94b22bd99b432722256da2bea1c4ef7fc2fe23b8d6a"}
Sep 15 21:28:57.744654 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:57.744608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85f847c588-ws6zl" event={"ID":"b80d04e9-56f1-49c7-801c-adfa3daad9f2","Type":"ContainerStarted","Data":"df72cbc22b75938c2b624dcdc0e5b6d6a7d940e520d062adcdcd953f4490fc74"}
Sep 15 21:28:57.759569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:57.759511    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85f847c588-ws6zl" podStartSLOduration=1.753406526 podStartE2EDuration="2.759490337s" totalImagesPullingTime="1.006083811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:57.757758082 +0000 UTC m=+766.158731328" watchObservedRunningTime="2026-09-15 21:28:57.759490337 +0000 UTC m=+766.160463605"
Sep 15 21:28:57.781332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:57.781287    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:28:57.781558 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:57.781526    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-747c668c6f-s2brw" podUID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" containerName="authorino" containerID="cri-o://c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41" gracePeriod=30
Sep 15 21:28:58.037140 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.037116    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:28:58.156450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.156406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7nn8\" (UniqueName: \"kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8\") pod \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") "
Sep 15 21:28:58.156450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.156448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert\") pod \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\" (UID: \"f8a0eeac-3b26-4f30-8559-efb9fad0e583\") "
Sep 15 21:28:58.158703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.158654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8" (OuterVolumeSpecName: "kube-api-access-l7nn8") pod "f8a0eeac-3b26-4f30-8559-efb9fad0e583" (UID: "f8a0eeac-3b26-4f30-8559-efb9fad0e583"). InnerVolumeSpecName "kube-api-access-l7nn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:58.167431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.167394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f8a0eeac-3b26-4f30-8559-efb9fad0e583" (UID: "f8a0eeac-3b26-4f30-8559-efb9fad0e583"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:58.258276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.258230    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nn8\" (UniqueName: \"kubernetes.io/projected/f8a0eeac-3b26-4f30-8559-efb9fad0e583-kube-api-access-l7nn8\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:28:58.258276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.258279    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8a0eeac-3b26-4f30-8559-efb9fad0e583-tls-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:28:58.751113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.751075    2578 generic.go:356] "Generic (PLEG): container finished" podID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" containerID="c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41" exitCode=0
Sep 15 21:28:58.751550 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.751136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-747c668c6f-s2brw"
Sep 15 21:28:58.751550 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.751161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747c668c6f-s2brw" event={"ID":"f8a0eeac-3b26-4f30-8559-efb9fad0e583","Type":"ContainerDied","Data":"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"}
Sep 15 21:28:58.751550 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.751216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-747c668c6f-s2brw" event={"ID":"f8a0eeac-3b26-4f30-8559-efb9fad0e583","Type":"ContainerDied","Data":"37f22c7984b1ef4e40a51cc012c80d207705bd928bcb95e3856f7b5cb140d479"}
Sep 15 21:28:58.751550 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.751238    2578 scope.go:117] "RemoveContainer" containerID="c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"
Sep 15 21:28:58.761014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.760994    2578 scope.go:117] "RemoveContainer" containerID="c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"
Sep 15 21:28:58.761408 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:28:58.761389    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41\": container with ID starting with c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41 not found: ID does not exist" containerID="c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"
Sep 15 21:28:58.761464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.761418    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41"} err="failed to get container status \"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41\": rpc error: code = NotFound desc = could not find container \"c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41\": container with ID starting with c64ab0bb48b1464271518c01c3bf17b196027af3d458eb543086178aab3f2b41 not found: ID does not exist"
Sep 15 21:28:58.784380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.784342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:28:58.800780 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:28:58.800735    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-747c668c6f-s2brw"]
Sep 15 21:29:00.075887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:00.075852    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" path="/var/lib/kubelet/pods/f8a0eeac-3b26-4f30-8559-efb9fad0e583/volumes"
Sep 15 21:29:16.995142 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:16.995110    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7763-db6fd2"]
Sep 15 21:29:16.995549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:16.995497    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" containerName="authorino"
Sep 15 21:29:16.995549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:16.995508    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" containerName="authorino"
Sep 15 21:29:16.995621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:16.995606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a0eeac-3b26-4f30-8559-efb9fad0e583" containerName="authorino"
Sep 15 21:29:17.000078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.000060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:17.001916 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.001892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\""
Sep 15 21:29:17.005607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.005577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7763-db6fd2"]
Sep 15 21:29:17.145679 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.145644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm8t\" (UniqueName: \"kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t\") pod \"test-curl-7763-db6fd2\" (UID: \"44d8d200-b8de-466d-8270-3623d397b231\") " pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:17.246604 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.246504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm8t\" (UniqueName: \"kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t\") pod \"test-curl-7763-db6fd2\" (UID: \"44d8d200-b8de-466d-8270-3623d397b231\") " pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:17.255379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.255352    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm8t\" (UniqueName: \"kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t\") pod \"test-curl-7763-db6fd2\" (UID: \"44d8d200-b8de-466d-8270-3623d397b231\") " pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:17.311248 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.311200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:17.583067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.583034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7763-db6fd2"]
Sep 15 21:29:17.584989 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:17.584960    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d8d200_b8de_466d_8270_3623d397b231.slice/crio-12bbf5210f76190d9c2517046ca0ac947f96d66a5ffd7bcbf14db64c3c1e222e WatchSource:0}: Error finding container 12bbf5210f76190d9c2517046ca0ac947f96d66a5ffd7bcbf14db64c3c1e222e: Status 404 returned error can't find the container with id 12bbf5210f76190d9c2517046ca0ac947f96d66a5ffd7bcbf14db64c3c1e222e
Sep 15 21:29:17.853076 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:17.852979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7763-db6fd2" event={"ID":"44d8d200-b8de-466d-8270-3623d397b231","Type":"ContainerStarted","Data":"12bbf5210f76190d9c2517046ca0ac947f96d66a5ffd7bcbf14db64c3c1e222e"}
Sep 15 21:29:19.754354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.754312    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:29:19.764006 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.763976    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.769072 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.769046    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b99d7b-serving-cert\""
Sep 15 21:29:19.771471 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.771378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.771628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.771520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.771698 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.771631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jcmh\" (UniqueName: \"kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.771809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.771743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.777696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.777630    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:29:19.873183 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.873149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jcmh\" (UniqueName: \"kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.873375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.873241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.873375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.873350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.873501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.873408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.873686 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:19.873489    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-serving-cert: secret "maas-api-e2e-worker-w0-b99d7b-serving-cert" not found
Sep 15 21:29:19.873954 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:19.873937    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls podName:356859e8-5908-4d4b-a2bc-8a9c204d7418 nodeName:}" failed. No retries permitted until 2026-09-15 21:29:20.373915497 +0000 UTC m=+788.774888742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls") pod "maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" (UID: "356859e8-5908-4d4b-a2bc-8a9c204d7418") : secret "maas-api-e2e-worker-w0-b99d7b-serving-cert" not found
Sep 15 21:29:19.874192 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.874149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.876113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.876091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:19.885059 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:19.885034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jcmh\" (UniqueName: \"kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:20.378746 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.378670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:20.381314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.381282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:20.677845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.677758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:20.696849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.696820    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:29:20.700550 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.700530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.702811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.702791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b99d7b\""
Sep 15 21:29:20.702936 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.702872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b99d7b-dockercfg-snttw\""
Sep 15 21:29:20.715873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.715846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:29:20.782484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.782453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.782908 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.782512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhxm\" (UniqueName: \"kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.871728 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.871680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7763-db6fd2" event={"ID":"44d8d200-b8de-466d-8270-3623d397b231","Type":"ContainerStarted","Data":"f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd"}
Sep 15 21:29:20.883646 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.883606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.883810 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.883686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhxm\" (UniqueName: \"kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.884438 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.884420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.887189 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.887140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7763-db6fd2" podStartSLOduration=2.240905216 podStartE2EDuration="4.887127061s" totalImagesPullingTime="2.646221845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:20.885689644 +0000 UTC m=+789.286662891" watchObservedRunningTime="2026-09-15 21:29:20.887127061 +0000 UTC m=+789.288100307"
Sep 15 21:29:20.891866 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.891842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhxm\" (UniqueName: \"kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm\") pod \"payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:20.954210 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:20.954177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:29:20.955953 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:20.955927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356859e8_5908_4d4b_a2bc_8a9c204d7418.slice/crio-8a94880b5fdedb8d29b0d66949e8623ec8d03201af3b960c45c8cbea014fa30f WatchSource:0}: Error finding container 8a94880b5fdedb8d29b0d66949e8623ec8d03201af3b960c45c8cbea014fa30f: Status 404 returned error can't find the container with id 8a94880b5fdedb8d29b0d66949e8623ec8d03201af3b960c45c8cbea014fa30f
Sep 15 21:29:21.012143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.012108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:21.034423 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.034390    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:29:21.038952 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.038926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.040829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.040768    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4bc420-serving-cert\""
Sep 15 21:29:21.054409 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.053906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:29:21.090918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.086637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.090918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.086764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.090918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.086813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfqm\" (UniqueName: \"kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.090918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.086852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.188099 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.188057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.188280 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.188223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.188351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.188288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.188351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.188320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfqm\" (UniqueName: \"kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.189150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.189115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.191070 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.191040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.191154 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.191141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.197519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.197494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfqm\" (UniqueName: \"kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm\") pod \"maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.330987 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.330960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:29:21.334419 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:21.334361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d6935b_0628_428a_909d_832e3d53d826.slice/crio-bbfd59823292ff5deaee1ad6699f0557ea87b42ee9316806214ad8cf69657b00 WatchSource:0}: Error finding container bbfd59823292ff5deaee1ad6699f0557ea87b42ee9316806214ad8cf69657b00: Status 404 returned error can't find the container with id bbfd59823292ff5deaee1ad6699f0557ea87b42ee9316806214ad8cf69657b00
Sep 15 21:29:21.386632 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.386607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:21.599566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.599470    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7763-db6fd2"]
Sep 15 21:29:21.618841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.618809    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7763-db6fd2"]
Sep 15 21:29:21.700262 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.700238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:29:21.702836 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:21.702808    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd271b92_b2ab_4e39_a393_8e71066c82f1.slice/crio-761c546847bb9065dc706408df91a19edf170d10fde79f23d50d1f84fd47de46 WatchSource:0}: Error finding container 761c546847bb9065dc706408df91a19edf170d10fde79f23d50d1f84fd47de46: Status 404 returned error can't find the container with id 761c546847bb9065dc706408df91a19edf170d10fde79f23d50d1f84fd47de46
Sep 15 21:29:21.878266 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.878163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" event={"ID":"356859e8-5908-4d4b-a2bc-8a9c204d7418","Type":"ContainerStarted","Data":"8a94880b5fdedb8d29b0d66949e8623ec8d03201af3b960c45c8cbea014fa30f"}
Sep 15 21:29:21.879795 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.879545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" event={"ID":"bd271b92-b2ab-4e39-a393-8e71066c82f1","Type":"ContainerStarted","Data":"761c546847bb9065dc706408df91a19edf170d10fde79f23d50d1f84fd47de46"}
Sep 15 21:29:21.881695 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.881666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" event={"ID":"69d6935b-0628-428a-909d-832e3d53d826","Type":"ContainerStarted","Data":"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"}
Sep 15 21:29:21.881855 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.881711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" event={"ID":"69d6935b-0628-428a-909d-832e3d53d826","Type":"ContainerStarted","Data":"bbfd59823292ff5deaee1ad6699f0557ea87b42ee9316806214ad8cf69657b00"}
Sep 15 21:29:21.881855 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.881772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:21.881855 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:21.881820    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7763-db6fd2" container="test-curl-7763-db6fd2"
Sep 15 21:29:21.930168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:21.930113    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" podStartSLOduration=1.93009372 podStartE2EDuration="1.93009372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:21.927581478 +0000 UTC m=+790.328554724" watchObservedRunningTime="2026-09-15 21:29:21.93009372 +0000 UTC m=+790.331066985"
Sep 15 21:29:22.317912 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.317878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:29:22.318299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.318287    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d8d200-b8de-466d-8270-3623d397b231" containerName="test-curl-7763-db6fd2"
Sep 15 21:29:22.318346 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.318301    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d8d200-b8de-466d-8270-3623d397b231" containerName="test-curl-7763-db6fd2"
Sep 15 21:29:22.318403 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.318394    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d8d200-b8de-466d-8270-3623d397b231" containerName="test-curl-7763-db6fd2"
Sep 15 21:29:22.322601 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.322584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.324706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.324686    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4bc420\""
Sep 15 21:29:22.333645 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.333615    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:29:22.398611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.398581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27c9\" (UniqueName: \"kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.398745 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.398641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.500121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.500078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27c9\" (UniqueName: \"kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.500324 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.500158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.501177 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.501153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.508189 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.508164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27c9\" (UniqueName: \"kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9\") pod \"payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.635776 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.635678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:22.889476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.889374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" event={"ID":"356859e8-5908-4d4b-a2bc-8a9c204d7418","Type":"ContainerStarted","Data":"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"}
Sep 15 21:29:22.889476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.889475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:22.891147 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.891111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" event={"ID":"bd271b92-b2ab-4e39-a393-8e71066c82f1","Type":"ContainerStarted","Data":"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"}
Sep 15 21:29:22.891264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.891172    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:22.891264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.891189    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7763-db6fd2" podUID="44d8d200-b8de-466d-8270-3623d397b231" containerName="test-curl-7763-db6fd2" containerID="cri-o://f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd" gracePeriod=2
Sep 15 21:29:22.910692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.910643    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" podStartSLOduration=2.481933189 podStartE2EDuration="3.910626002s" totalImagesPullingTime="1.428692813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:22.908327356 +0000 UTC m=+791.309300606" watchObservedRunningTime="2026-09-15 21:29:22.910626002 +0000 UTC m=+791.311599247"
Sep 15 21:29:22.923286 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.923083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:29:22.925611 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:22.925583    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228752dd_c92f_4307_9977_4fd9a38f880a.slice/crio-a629fc402fbe94b46256e89f4d5e7db4d72c51adf976ab5fa628dcb30b85cd3c WatchSource:0}: Error finding container a629fc402fbe94b46256e89f4d5e7db4d72c51adf976ab5fa628dcb30b85cd3c: Status 404 returned error can't find the container with id a629fc402fbe94b46256e89f4d5e7db4d72c51adf976ab5fa628dcb30b85cd3c
Sep 15 21:29:22.930666 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:22.930609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" podStartSLOduration=1.223702503 podStartE2EDuration="1.930588317s" totalImagesPullingTime="706.885814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:22.929796042 +0000 UTC m=+791.330769287" watchObservedRunningTime="2026-09-15 21:29:22.930588317 +0000 UTC m=+791.331561566"
Sep 15 21:29:23.898944 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:23.898891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" event={"ID":"228752dd-c92f-4307-9977-4fd9a38f880a","Type":"ContainerStarted","Data":"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"}
Sep 15 21:29:23.899408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:23.898959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" event={"ID":"228752dd-c92f-4307-9977-4fd9a38f880a","Type":"ContainerStarted","Data":"a629fc402fbe94b46256e89f4d5e7db4d72c51adf976ab5fa628dcb30b85cd3c"}
Sep 15 21:29:23.899408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:23.899170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:23.915579 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:23.915532    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" podStartSLOduration=1.915516673 podStartE2EDuration="1.915516673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:23.913519743 +0000 UTC m=+792.314492984" watchObservedRunningTime="2026-09-15 21:29:23.915516673 +0000 UTC m=+792.316489919"
Sep 15 21:29:25.044526 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.044490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:25.057708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.057678    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:29:25.061176 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.061152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.063417 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.063397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-10c104-openshift-default-dockercfg-rmf7s\""
Sep 15 21:29:25.063546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.063530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:29:25.078081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.078051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:29:25.126415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jm8t\" (UniqueName: \"kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t\") pod \"44d8d200-b8de-466d-8270-3623d397b231\" (UID: \"44d8d200-b8de-466d-8270-3623d397b231\") "
Sep 15 21:29:25.126593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tlx\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126746 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126780 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.126900 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.126884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.128564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.128542    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t" (OuterVolumeSpecName: "kube-api-access-9jm8t") pod "44d8d200-b8de-466d-8270-3623d397b231" (UID: "44d8d200-b8de-466d-8270-3623d397b231"). InnerVolumeSpecName "kube-api-access-9jm8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:25.228135 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tlx\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228456    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jm8t\" (UniqueName: \"kubernetes.io/projected/44d8d200-b8de-466d-8270-3623d397b231-kube-api-access-9jm8t\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:29:25.228773 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228924 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.228976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.228918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.229080 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.229059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.230575 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.230545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.230884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.230864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.241019 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.240995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.241130 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.240998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tlx\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx\") pod \"e2e-autores-10c104-openshift-default-58f575cdcb-gppd2\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") " pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.371511 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.371472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:25.660389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.660347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:29:25.663250 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:25.663213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d099ef_2cd3_4d31_a5f0_6868aa94c0f0.slice/crio-0fa7b137e27d5c2c0f4ef93c900f52d9e5e6851ad651099d0af5fb2cbfdbca2b WatchSource:0}: Error finding container 0fa7b137e27d5c2c0f4ef93c900f52d9e5e6851ad651099d0af5fb2cbfdbca2b: Status 404 returned error can't find the container with id 0fa7b137e27d5c2c0f4ef93c900f52d9e5e6851ad651099d0af5fb2cbfdbca2b
Sep 15 21:29:25.740015 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.739983    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:29:25.744781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.744753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.746896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.746871    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ac4411-serving-cert\""
Sep 15 21:29:25.760693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.760656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:29:25.834191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.834156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.834356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.834266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpcs4\" (UniqueName: \"kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.834356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.834301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.834501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.834402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.914489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.914407    2578 generic.go:356] "Generic (PLEG): container finished" podID="44d8d200-b8de-466d-8270-3623d397b231" containerID="f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd" exitCode=137
Sep 15 21:29:25.914489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.914445    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:25.914705 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.914498    2578 scope.go:117] "RemoveContainer" containerID="f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd"
Sep 15 21:29:25.915907 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.915738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerStarted","Data":"0fa7b137e27d5c2c0f4ef93c900f52d9e5e6851ad651099d0af5fb2cbfdbca2b"}
Sep 15 21:29:25.923676 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.923654    2578 scope.go:117] "RemoveContainer" containerID="f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd"
Sep 15 21:29:25.923961 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:25.923937    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd\": container with ID starting with f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd not found: ID does not exist" containerID="f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd"
Sep 15 21:29:25.924041 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.923977    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd"} err="failed to get container status \"f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd\": rpc error: code = NotFound desc = could not find container \"f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd\": container with ID starting with f4924d4eba11cebac556ea4ba65a53ec24bf2b2c0f5c7de4f1e2ae515e59cafd not found: ID does not exist"
Sep 15 21:29:25.924445 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:25.924426    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7763-db6fd2\" is forbidden: User \"system:node:ip-10-0-135-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-105.ec2.internal' and this object" podUID="44d8d200-b8de-466d-8270-3623d397b231" pod="opendatahub/test-curl-7763-db6fd2"
Sep 15 21:29:25.935770 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.935739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.935860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.935791    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.935860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.935834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.935932 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.935917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpcs4\" (UniqueName: \"kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.936513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.936492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.938093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.938063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.938197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.938104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:25.943071 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:25.943048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpcs4\" (UniqueName: \"kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4\") pod \"maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:26.057033 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.056997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:26.076339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.076309    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d8d200-b8de-466d-8270-3623d397b231" path="/var/lib/kubelet/pods/44d8d200-b8de-466d-8270-3623d397b231/volumes"
Sep 15 21:29:26.338531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.338506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:29:26.341197 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:26.341171    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efb254f_4201_473e_ac54_62485172d790.slice/crio-9e3fe23b729ddc2936fd83e18989e63048301100b77be8748cb15b0263117cb9 WatchSource:0}: Error finding container 9e3fe23b729ddc2936fd83e18989e63048301100b77be8748cb15b0263117cb9: Status 404 returned error can't find the container with id 9e3fe23b729ddc2936fd83e18989e63048301100b77be8748cb15b0263117cb9
Sep 15 21:29:26.926637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.926591    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" event={"ID":"4efb254f-4201-473e-ac54-62485172d790","Type":"ContainerStarted","Data":"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"}
Sep 15 21:29:26.926842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.926821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:26.926945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:26.926867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" event={"ID":"4efb254f-4201-473e-ac54-62485172d790","Type":"ContainerStarted","Data":"9e3fe23b729ddc2936fd83e18989e63048301100b77be8748cb15b0263117cb9"}
Sep 15 21:29:28.517661 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.517622    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:28.517960 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.517694    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:28.517960 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.517773    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:28.906485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.906408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:29:28.906627 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.906501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:29:28.930429 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.930370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" podStartSLOduration=3.593315782 podStartE2EDuration="3.930352687s" totalImagesPullingTime="337.036905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:26.954097141 +0000 UTC m=+795.355070392" watchObservedRunningTime="2026-09-15 21:29:28.930352687 +0000 UTC m=+797.331325932"
Sep 15 21:29:28.944996 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:28.944957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerStarted","Data":"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"}
Sep 15 21:29:29.016600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:29.016533    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podStartSLOduration=1.165210473 podStartE2EDuration="4.016510403s" totalImagesPullingTime="2.85129993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:29.014674447 +0000 UTC m=+797.415647693" watchObservedRunningTime="2026-09-15 21:29:29.016510403 +0000 UTC m=+797.417483655"
Sep 15 21:29:29.371799 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:29.371763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:29.376420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:29.376394    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:29.950587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:29.950555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:29.951625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:29.951606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:31.240451 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.240414    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:29:31.244377 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.244355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.246556 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.246534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c3e1d9c1-serving-cert\""
Sep 15 21:29:31.261079 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.261057    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:29:31.294728 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.294684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkjn\" (UniqueName: \"kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.294897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.294797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.294897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.294840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.294897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.294870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.395630 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.395593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.395630 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.395636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.395897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.395755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.395897 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:31.395761    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-serving-cert: secret "maas-api-e2e-ait-c3e1d9c1-serving-cert" not found
Sep 15 21:29:31.395897 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:31.395855    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls podName:6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80 nodeName:}" failed. No retries permitted until 2026-09-15 21:29:31.895832973 +0000 UTC m=+800.296806200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls") pod "maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" (UID: "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80") : secret "maas-api-e2e-ait-c3e1d9c1-serving-cert" not found
Sep 15 21:29:31.396029 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.395911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkjn\" (UniqueName: \"kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.396353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.396330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.398309 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.398286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.406771 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.406749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkjn\" (UniqueName: \"kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.901538 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.901499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:31.903895 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:31.903875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") pod \"maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:32.154962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.154873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:32.439210 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.439170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:29:32.441991 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:32.441959    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7ea10d_f59c_4b15_b09b_4ee01a7d5e80.slice/crio-b028c9da126b4cfaf1436a9b702cfcf1c9b19b0b5c1a64125c8e99a50d8c3107 WatchSource:0}: Error finding container b028c9da126b4cfaf1436a9b702cfcf1c9b19b0b5c1a64125c8e99a50d8c3107: Status 404 returned error can't find the container with id b028c9da126b4cfaf1436a9b702cfcf1c9b19b0b5c1a64125c8e99a50d8c3107
Sep 15 21:29:32.643587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.643553    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:29:32.648696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.648660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.651021 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.650996    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-10c104-serving-cert\""
Sep 15 21:29:32.667277 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.667253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:29:32.708951 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.708855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.708951 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.708893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvxp\" (UniqueName: \"kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.709209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.709127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.709209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.709168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.810689 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.810640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.810689 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.810693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.810930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.810750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.810930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.810767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvxp\" (UniqueName: \"kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.810930 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:32.810892    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-10c104-serving-cert: secret "maas-api-e2e-autores-10c104-serving-cert" not found
Sep 15 21:29:32.811026 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:32.810958    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls podName:8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1 nodeName:}" failed. No retries permitted until 2026-09-15 21:29:33.31093975 +0000 UTC m=+801.711912995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls") pod "maas-api-e2e-autores-10c104-6b6c8d7744-778mj" (UID: "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1") : secret "maas-api-e2e-autores-10c104-serving-cert" not found
Sep 15 21:29:32.811443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.811414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.813289 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.813266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.819551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.819525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvxp\" (UniqueName: \"kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:32.893253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.893223    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:29:32.939960 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.939933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:29:32.977335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:32.977242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" event={"ID":"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80","Type":"ContainerStarted","Data":"b028c9da126b4cfaf1436a9b702cfcf1c9b19b0b5c1a64125c8e99a50d8c3107"}
Sep 15 21:29:33.314978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.314883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:33.317368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.317326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") pod \"maas-api-e2e-autores-10c104-6b6c8d7744-778mj\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:33.491848 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.491811    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:29:33.495911 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.495889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.498271 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.498252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-10c104\""
Sep 15 21:29:33.498368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.498256    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-10c104-dockercfg-gnkwf\""
Sep 15 21:29:33.513984 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.513956    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:29:33.561051 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.561017    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:33.619807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.618434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdtj\" (UniqueName: \"kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.619807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.618493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.720098 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.720053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdtj\" (UniqueName: \"kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.720098 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.720104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.720891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.720871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.732475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.732443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdtj\" (UniqueName: \"kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj\") pod \"payload-processing-e2e-autores-10c104-7b6556b647-rnc95\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") " pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.806557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.806518    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:33.848000 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.847922    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:29:33.849740 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:33.849688    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8723ac7c_edf4_4eb7_8d4a_0a6f28c3b3c1.slice/crio-dce28ab09e2b3ff3d3f83f45fe9166b56709d7a82f2606e6f32d3ca59f789513 WatchSource:0}: Error finding container dce28ab09e2b3ff3d3f83f45fe9166b56709d7a82f2606e6f32d3ca59f789513: Status 404 returned error can't find the container with id dce28ab09e2b3ff3d3f83f45fe9166b56709d7a82f2606e6f32d3ca59f789513
Sep 15 21:29:33.983812 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:33.983774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" event={"ID":"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1","Type":"ContainerStarted","Data":"dce28ab09e2b3ff3d3f83f45fe9166b56709d7a82f2606e6f32d3ca59f789513"}
Sep 15 21:29:34.116128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.116105    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:29:34.118393 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:29:34.118364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f8bfd7_6ba0_4719_9b2b_ffcf28a46207.slice/crio-6e1b2fffacb31fea9ea3d8deb64bacc88d02a2f7d134c80514406f50fccf5823 WatchSource:0}: Error finding container 6e1b2fffacb31fea9ea3d8deb64bacc88d02a2f7d134c80514406f50fccf5823: Status 404 returned error can't find the container with id 6e1b2fffacb31fea9ea3d8deb64bacc88d02a2f7d134c80514406f50fccf5823
Sep 15 21:29:34.910247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.910218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:29:34.992331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.992295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" event={"ID":"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1","Type":"ContainerStarted","Data":"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"}
Sep 15 21:29:34.992485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.992368    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:34.994398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.994269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" event={"ID":"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207","Type":"ContainerStarted","Data":"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"}
Sep 15 21:29:34.994398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.994330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" event={"ID":"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207","Type":"ContainerStarted","Data":"6e1b2fffacb31fea9ea3d8deb64bacc88d02a2f7d134c80514406f50fccf5823"}
Sep 15 21:29:34.994398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:34.994364    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:35.008944 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:35.008890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" podStartSLOduration=2.6016048290000002 podStartE2EDuration="3.008869693s" totalImagesPullingTime="407.264864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:35.007913958 +0000 UTC m=+803.408887204" watchObservedRunningTime="2026-09-15 21:29:35.008869693 +0000 UTC m=+803.409842944"
Sep 15 21:29:35.027958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:35.027900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" podStartSLOduration=2.027882979 podStartE2EDuration="2.027882979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:35.026354781 +0000 UTC m=+803.427328035" watchObservedRunningTime="2026-09-15 21:29:35.027882979 +0000 UTC m=+803.428856224"
Sep 15 21:29:36.008458 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:36.008414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" event={"ID":"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80","Type":"ContainerStarted","Data":"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"}
Sep 15 21:29:36.008979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:36.008915    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:36.028626 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:36.028570    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" podStartSLOduration=2.543001451 podStartE2EDuration="5.028553317s" totalImagesPullingTime="2.485551866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:36.026400937 +0000 UTC m=+804.427374184" watchObservedRunningTime="2026-09-15 21:29:36.028553317 +0000 UTC m=+804.429526599"
Sep 15 21:29:41.014358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:41.014329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:29:43.024329 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:43.024298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:46.010860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.010830    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:29:46.379792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.374862    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:29:46.386558 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.382258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.390946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.388340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bb7867-dockercfg-wvj2r\""
Sep 15 21:29:46.390946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.388586    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bb7867\""
Sep 15 21:29:46.397397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.392969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:29:46.455631 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.455595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdkp\" (UniqueName: \"kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.455822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.455661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.557264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.557224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdkp\" (UniqueName: \"kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.557473 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.557290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.558008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.557989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.575909 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.575870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdkp\" (UniqueName: \"kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp\") pod \"payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.703141 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.703097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:46.981566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:46.981532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:29:47.082867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:47.081801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" event={"ID":"52ac5b02-9cc5-4ea6-bff7-f17281469e7f","Type":"ContainerStarted","Data":"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"}
Sep 15 21:29:47.082867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:47.081875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" event={"ID":"52ac5b02-9cc5-4ea6-bff7-f17281469e7f","Type":"ContainerStarted","Data":"bb7199e731c3ed5458612f7791b86210d36f5152e1661dddb3b9b93aa349d842"}
Sep 15 21:29:47.082867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:47.082799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:47.120686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:47.120625    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" podStartSLOduration=1.120605667 podStartE2EDuration="1.120605667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:29:47.117385016 +0000 UTC m=+815.518358262" watchObservedRunningTime="2026-09-15 21:29:47.120605667 +0000 UTC m=+815.521578953"
Sep 15 21:29:51.107531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.107498    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerID="5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577" exitCode=0
Sep 15 21:29:51.108011 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.107574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerDied","Data":"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"}
Sep 15 21:29:51.108011 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.107993    2578 scope.go:117] "RemoveContainer" containerID="5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"
Sep 15 21:29:51.109128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.109094    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:51.109242 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.109202    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:51.109306 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:51.109249    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:29:52.114802 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:52.114765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerStarted","Data":"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"}
Sep 15 21:29:52.372051 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:52.371958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:52.372208 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:52.372149    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:52.372322 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:52.372219    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:53.372537 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:53.372488    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:53.373091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:53.372560    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:54.372911 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:54.372877    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:54.373529 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:54.373495    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:55.372865 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:55.372826    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:55.373541 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:55.373137    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:56.372657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:56.372601    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:56.372902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:56.372683    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:57.371993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:57.371955    2578 patch_prober.go:28] interesting pod/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 15 21:29:57.373692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:57.373656    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 15 21:29:58.055891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.055843    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:29:58.056401 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.056367    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" podUID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" containerName="maas-api" containerID="cri-o://3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603" gracePeriod=30
Sep 15 21:29:58.334002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.333916    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:58.377581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.377552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:58.378079 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.377992    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:58.378792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.378770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:29:58.487210 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.487182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") pod \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") "
Sep 15 21:29:58.487407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.487235    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbkjn\" (UniqueName: \"kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn\") pod \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") "
Sep 15 21:29:58.487540 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.487442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca\") pod \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") "
Sep 15 21:29:58.487540 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.487518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs\") pod \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\" (UID: \"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80\") "
Sep 15 21:29:58.487977 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.487939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" (UID: "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:29:58.488419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.488382    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:29:58.491156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.491133    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" (UID: "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:58.492611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.492581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" (UID: "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:29:58.492752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.492709    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn" (OuterVolumeSpecName: "kube-api-access-hbkjn") pod "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" (UID: "6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80"). InnerVolumeSpecName "kube-api-access-hbkjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:29:58.589043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.588955    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:29:58.589043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.588988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbkjn\" (UniqueName: \"kubernetes.io/projected/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-kube-api-access-hbkjn\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:29:58.589043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:58.588999    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:29:59.096469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.096436    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:29:59.156686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.156646    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" containerID="3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603" exitCode=0
Sep 15 21:29:59.156898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.156698    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"
Sep 15 21:29:59.156898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.156759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" event={"ID":"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80","Type":"ContainerDied","Data":"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"}
Sep 15 21:29:59.156898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.156815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9" event={"ID":"6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80","Type":"ContainerDied","Data":"b028c9da126b4cfaf1436a9b702cfcf1c9b19b0b5c1a64125c8e99a50d8c3107"}
Sep 15 21:29:59.156898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.156848    2578 scope.go:117] "RemoveContainer" containerID="3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"
Sep 15 21:29:59.166836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.166814    2578 scope.go:117] "RemoveContainer" containerID="3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"
Sep 15 21:29:59.167120 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:29:59.167098    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603\": container with ID starting with 3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603 not found: ID does not exist" containerID="3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"
Sep 15 21:29:59.167231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.167128    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603"} err="failed to get container status \"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603\": rpc error: code = NotFound desc = could not find container \"3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603\": container with ID starting with 3fffb809bf3eac93a462c0b9af0512550a84c13d1008ec67ac617e5172ab0603 not found: ID does not exist"
Sep 15 21:29:59.180991 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.180966    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:29:59.192885 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:29:59.192857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3e1d9c1-5d7dc8547-gctb9"]
Sep 15 21:30:00.075773 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:00.075744    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" path="/var/lib/kubelet/pods/6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80/volumes"
Sep 15 21:30:04.335461 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.335425    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:04.335988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.335959    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" containerName="maas-api"
Sep 15 21:30:04.335988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.335972    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" containerName="maas-api"
Sep 15 21:30:04.336128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.336047    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7ea10d-f59c-4b15-b09b-4ee01a7d5e80" containerName="maas-api"
Sep 15 21:30:04.340223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.340198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.342443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.342424    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5a03be06-serving-cert\""
Sep 15 21:30:04.350345 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.350320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzcq\" (UniqueName: \"kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.350450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.350390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.350450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.350422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.350533 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.350444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.355380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.355358    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:04.451387 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.451355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzcq\" (UniqueName: \"kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.451581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.451420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.451581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.451443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.451581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.451463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.451711 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:30:04.451575    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-serving-cert: secret "maas-api-e2e-migrate-5a03be06-serving-cert" not found
Sep 15 21:30:04.451711 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:30:04.451643    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls podName:1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c nodeName:}" failed. No retries permitted until 2026-09-15 21:30:04.95162227 +0000 UTC m=+833.352595513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls") pod "maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" (UID: "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c") : secret "maas-api-e2e-migrate-5a03be06-serving-cert" not found
Sep 15 21:30:04.452142 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.452118    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.453811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.453791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.464981 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.464961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzcq\" (UniqueName: \"kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.957179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.957141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:04.959564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:04.959536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") pod \"maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:05.250871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:05.250762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:05.543134 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:05.543112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:05.545916 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:30:05.545889    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab3bb4c_9a34_4a2c_bf3e_1227cb92093c.slice/crio-ea8e11b5cc3a8ae1f0adde44b068b2b05dccbf5c44bbc13f3f6e418f59bc49f9 WatchSource:0}: Error finding container ea8e11b5cc3a8ae1f0adde44b068b2b05dccbf5c44bbc13f3f6e418f59bc49f9: Status 404 returned error can't find the container with id ea8e11b5cc3a8ae1f0adde44b068b2b05dccbf5c44bbc13f3f6e418f59bc49f9
Sep 15 21:30:06.197274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:06.197231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" event={"ID":"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c","Type":"ContainerStarted","Data":"0f7a2c1626aa82b9e16e2d1691f913a5e9a5995a848a2266f1157f6cafcefc1f"}
Sep 15 21:30:06.197451 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:06.197286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" event={"ID":"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c","Type":"ContainerStarted","Data":"ea8e11b5cc3a8ae1f0adde44b068b2b05dccbf5c44bbc13f3f6e418f59bc49f9"}
Sep 15 21:30:06.197451 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:06.197416    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:06.243551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:06.243496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" podStartSLOduration=1.937864785 podStartE2EDuration="2.243480585s" totalImagesPullingTime="305.6158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:06.237086182 +0000 UTC m=+834.638059428" watchObservedRunningTime="2026-09-15 21:30:06.243480585 +0000 UTC m=+834.644453831"
Sep 15 21:30:12.207854 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:12.207808    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:34.200594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.200558    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:34.201100 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.200843    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" podUID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" containerName="maas-api" containerID="cri-o://0f7a2c1626aa82b9e16e2d1691f913a5e9a5995a848a2266f1157f6cafcefc1f" gracePeriod=30
Sep 15 21:30:34.354611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.354581    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" containerID="0f7a2c1626aa82b9e16e2d1691f913a5e9a5995a848a2266f1157f6cafcefc1f" exitCode=0
Sep 15 21:30:34.354791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.354637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" event={"ID":"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c","Type":"ContainerDied","Data":"0f7a2c1626aa82b9e16e2d1691f913a5e9a5995a848a2266f1157f6cafcefc1f"}
Sep 15 21:30:34.457190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.457126    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:34.562439 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.562400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") pod \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") "
Sep 15 21:30:34.562648 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.562509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs\") pod \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") "
Sep 15 21:30:34.562989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.562967    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlzcq\" (UniqueName: \"kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq\") pod \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") "
Sep 15 21:30:34.563082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.563047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca\") pod \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\" (UID: \"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c\") "
Sep 15 21:30:34.563908 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.563877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" (UID: "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:30:34.565677 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.565638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" (UID: "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:34.567419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.567380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" (UID: "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:30:34.568774 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.568709    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq" (OuterVolumeSpecName: "kube-api-access-hlzcq") pod "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" (UID: "1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c"). InnerVolumeSpecName "kube-api-access-hlzcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:30:34.665268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.665229    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.665268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.665270    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.665497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.665286    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlzcq\" (UniqueName: \"kubernetes.io/projected/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-kube-api-access-hlzcq\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:30:34.665497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:34.665301    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:30:35.362548 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:35.362505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz" event={"ID":"1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c","Type":"ContainerDied","Data":"ea8e11b5cc3a8ae1f0adde44b068b2b05dccbf5c44bbc13f3f6e418f59bc49f9"}
Sep 15 21:30:35.363083 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:35.362547    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"
Sep 15 21:30:35.363083 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:35.362587    2578 scope.go:117] "RemoveContainer" containerID="0f7a2c1626aa82b9e16e2d1691f913a5e9a5995a848a2266f1157f6cafcefc1f"
Sep 15 21:30:35.389024 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:35.388969    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:35.394762 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:35.394732    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5a03be06-6b6555855f-9k2hz"]
Sep 15 21:30:36.076650 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:36.076619    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" path="/var/lib/kubelet/pods/1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c/volumes"
Sep 15 21:30:40.204353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.204315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:30:40.205124 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.205100    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" containerName="maas-api"
Sep 15 21:30:40.205124 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.205123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" containerName="maas-api"
Sep 15 21:30:40.205299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.205237    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ab3bb4c-9a34-4a2c-bf3e-1227cb92093c" containerName="maas-api"
Sep 15 21:30:40.208993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.208975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.212523 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.212500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfw5\" (UniqueName: \"kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.212631 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.212550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.212681 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.212636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.212836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.212699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.213030 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.213009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-df1389fe-serving-cert\""
Sep 15 21:30:40.223851 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.223824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:30:40.313511 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.313481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfw5\" (UniqueName: \"kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.313673 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.313523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.313673 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.313575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.313673 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.313633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.313823 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:30:40.313787    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-serving-cert: secret "maas-api-e2e-ait-df1389fe-serving-cert" not found
Sep 15 21:30:40.313909 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:30:40.313849    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls podName:3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca nodeName:}" failed. No retries permitted until 2026-09-15 21:30:40.813832726 +0000 UTC m=+869.214805950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls") pod "maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" (UID: "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca") : secret "maas-api-e2e-ait-df1389fe-serving-cert" not found
Sep 15 21:30:40.314324 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.314297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.316082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.316053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.322113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.322088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfw5\" (UniqueName: \"kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.818791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.818757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:40.821230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:40.821204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") pod \"maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:41.120531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:41.120451    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:41.408036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:41.408010    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:30:41.409348 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:30:41.409324    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3fcbc3_a420_4c8b_a7d3_cdb809ae31ca.slice/crio-4c9824e5e1f4aa974bb54b9da8c4a6385917f24294b4176927e369365274e5a3 WatchSource:0}: Error finding container 4c9824e5e1f4aa974bb54b9da8c4a6385917f24294b4176927e369365274e5a3: Status 404 returned error can't find the container with id 4c9824e5e1f4aa974bb54b9da8c4a6385917f24294b4176927e369365274e5a3
Sep 15 21:30:42.406998 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:42.406954    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" event={"ID":"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca","Type":"ContainerStarted","Data":"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"}
Sep 15 21:30:42.407186 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:42.407027    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:30:42.407186 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:42.407053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" event={"ID":"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca","Type":"ContainerStarted","Data":"4c9824e5e1f4aa974bb54b9da8c4a6385917f24294b4176927e369365274e5a3"}
Sep 15 21:30:42.429573 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:42.429519    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" podStartSLOduration=1.972398345 podStartE2EDuration="2.429504723s" totalImagesPullingTime="457.106378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:30:42.427976901 +0000 UTC m=+870.828950150" watchObservedRunningTime="2026-09-15 21:30:42.429504723 +0000 UTC m=+870.830477968"
Sep 15 21:30:48.418413 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:30:48.418376    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:31:09.833344 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:09.833245    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:31:09.833822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:09.833503    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" podUID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" containerName="maas-api" containerID="cri-o://ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e" gracePeriod=30
Sep 15 21:31:10.093735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.093656    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:31:10.224657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.224616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca\") pod \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") "
Sep 15 21:31:10.224657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.224660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svfw5\" (UniqueName: \"kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5\") pod \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") "
Sep 15 21:31:10.224914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.224747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") pod \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") "
Sep 15 21:31:10.224914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.224816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs\") pod \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\" (UID: \"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca\") "
Sep 15 21:31:10.225108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.225076    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" (UID: "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:31:10.225279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.225261    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:31:10.227002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.226972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5" (OuterVolumeSpecName: "kube-api-access-svfw5") pod "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" (UID: "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca"). InnerVolumeSpecName "kube-api-access-svfw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:31:10.227120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.227026    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" (UID: "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:10.227120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.227048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" (UID: "3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:31:10.326253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.326223    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:31:10.326253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.326251    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:31:10.326448 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.326262    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svfw5\" (UniqueName: \"kubernetes.io/projected/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca-kube-api-access-svfw5\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:31:10.555726 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.555691    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" containerID="ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e" exitCode=0
Sep 15 21:31:10.555904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.555750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"
Sep 15 21:31:10.555904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.555757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" event={"ID":"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca","Type":"ContainerDied","Data":"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"}
Sep 15 21:31:10.555904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.555807    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7" event={"ID":"3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca","Type":"ContainerDied","Data":"4c9824e5e1f4aa974bb54b9da8c4a6385917f24294b4176927e369365274e5a3"}
Sep 15 21:31:10.555904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.555833    2578 scope.go:117] "RemoveContainer" containerID="ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"
Sep 15 21:31:10.565560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.565542    2578 scope.go:117] "RemoveContainer" containerID="ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"
Sep 15 21:31:10.565864 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:31:10.565843    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e\": container with ID starting with ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e not found: ID does not exist" containerID="ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"
Sep 15 21:31:10.565941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.565873    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e"} err="failed to get container status \"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e\": rpc error: code = NotFound desc = could not find container \"ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e\": container with ID starting with ee9e0925b2c1781e30f323916c3731ed959b21e6bc8fbe1e8aecf75656c1b97e not found: ID does not exist"
Sep 15 21:31:10.577656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.577628    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:31:10.581052 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:10.581028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df1389fe-565dc69d4b-rnxl7"]
Sep 15 21:31:11.958417 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.958384    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:31:11.958915 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.958900    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" containerName="maas-api"
Sep 15 21:31:11.958915 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.958916    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" containerName="maas-api"
Sep 15 21:31:11.958996 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.958984    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" containerName="maas-api"
Sep 15 21:31:11.963530 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.963503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:11.966033 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.966007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d578a96c-openshift-default-dockercfg-8b9z2\""
Sep 15 21:31:11.977977 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:11.977952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:31:12.040491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m8s4\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.040950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.040795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.081833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.081799    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca" path="/var/lib/kubelet/pods/3b3fcbc3-a420-4c8b-a7d3-cdb809ae31ca/volumes"
Sep 15 21:31:12.142132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m8s4\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.142810 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.143225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.142977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.143225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.143014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.143225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.143053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.143225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.143177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.144605 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.144584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.144842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.144823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.151868 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.151844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.165488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.165453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m8s4\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4\") pod \"e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") " pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.274537 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.274443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:12.556593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.556568    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:31:12.558526 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:31:12.558500    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb78eb7_a2ec_4593_8f92_8b5cebf14618.slice/crio-d3b5d9e32c64d9d3172ed4ec15e5b1608512034361ec08b2103b000a32e5ced0 WatchSource:0}: Error finding container d3b5d9e32c64d9d3172ed4ec15e5b1608512034361ec08b2103b000a32e5ced0: Status 404 returned error can't find the container with id d3b5d9e32c64d9d3172ed4ec15e5b1608512034361ec08b2103b000a32e5ced0
Sep 15 21:31:12.561352 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.561315    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:12.561430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.561383    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:12.561430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.561412    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:12.568182 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:12.568158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerStarted","Data":"d3b5d9e32c64d9d3172ed4ec15e5b1608512034361ec08b2103b000a32e5ced0"}
Sep 15 21:31:13.575311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:13.575275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerStarted","Data":"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"}
Sep 15 21:31:13.595832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:13.595785    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podStartSLOduration=2.5957682650000002 podStartE2EDuration="2.595768265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:13.594449168 +0000 UTC m=+901.995422413" watchObservedRunningTime="2026-09-15 21:31:13.595768265 +0000 UTC m=+901.996741573"
Sep 15 21:31:14.275635 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:14.275594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:14.280812 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:14.280779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:14.580556 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:14.580474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:14.581593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:14.581575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:37.726440 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.726398    2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerID="4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5" exitCode=0
Sep 15 21:31:37.726873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.726473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerDied","Data":"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"}
Sep 15 21:31:37.726954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.726926    2578 scope.go:117] "RemoveContainer" containerID="4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"
Sep 15 21:31:37.728108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.728077    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:37.728202 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.728152    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:37.728202 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:37.728182    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:31:38.734424 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:38.734389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerStarted","Data":"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"}
Sep 15 21:31:39.275693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:39.275654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:39.276017 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:39.275984    2578 patch_prober.go:28] interesting pod/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 15 21:31:39.276253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:39.276224    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 15 21:31:40.275903 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:40.275852    2578 patch_prober.go:28] interesting pod/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 15 21:31:40.276436 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:40.275963    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 15 21:31:41.275164 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:41.275117    2578 patch_prober.go:28] interesting pod/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 15 21:31:41.275361 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:41.275205    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 15 21:31:42.275239 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:42.275203    2578 patch_prober.go:28] interesting pod/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 15 21:31:42.275844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:42.275809    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 15 21:31:43.275656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:43.275614    2578 patch_prober.go:28] interesting pod/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 15 21:31:43.276291 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:43.276258    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 15 21:31:44.280074 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:44.280039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:44.280478 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:44.280456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:44.281233 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:44.281212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:31:52.134458 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.134413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:31:52.141012 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.140980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.143235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.143212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4bc420\"/\"kube-root-ca.crt\""
Sep 15 21:31:52.143369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.143300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4bc420\"/\"test-model-4bc420-kserve-self-signed-certs\""
Sep 15 21:31:52.143481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.143464    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4bc420\"/\"openshift-service-ca.crt\""
Sep 15 21:31:52.144238 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.144215    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4bc420\"/\"default-dockercfg-xnkb8\""
Sep 15 21:31:52.150939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.150912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:31:52.197523 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.197523 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.197850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.197850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.197850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.197982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.197885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.298713 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.298906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.298906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.298906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.299053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.299053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.298951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.299314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.299289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.299389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.299348    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.299389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.299368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.301132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.301111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.301431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.301412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.306810 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.306784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf\") pod \"test-model-4bc420-kserve-8456d75955-plw49\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") " pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.456151 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.456111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:31:52.731155 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.731075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:31:52.734180 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:31:52.734144    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d01ae12_81d0_4d1b_935b_afc5c4018c2f.slice/crio-3d189e688941518e53e8eeb2b385c5a28629578957a716ee553dc2c6a9be0d40 WatchSource:0}: Error finding container 3d189e688941518e53e8eeb2b385c5a28629578957a716ee553dc2c6a9be0d40: Status 404 returned error can't find the container with id 3d189e688941518e53e8eeb2b385c5a28629578957a716ee553dc2c6a9be0d40
Sep 15 21:31:52.818135 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.818096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" event={"ID":"9d01ae12-81d0-4d1b-935b-afc5c4018c2f","Type":"ContainerStarted","Data":"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"}
Sep 15 21:31:52.818299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.818150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" event={"ID":"9d01ae12-81d0-4d1b-935b-afc5c4018c2f","Type":"ContainerStarted","Data":"3d189e688941518e53e8eeb2b385c5a28629578957a716ee553dc2c6a9be0d40"}
Sep 15 21:31:52.837553 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:52.837495    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" podStartSLOduration=0.837473285 podStartE2EDuration="837.473285ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:52.836571917 +0000 UTC m=+941.237545163" watchObservedRunningTime="2026-09-15 21:31:52.837473285 +0000 UTC m=+941.238446532"
Sep 15 21:31:59.771408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:59.771364    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:31:59.771965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:31:59.771629    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy" containerID="cri-o://6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2" gracePeriod=30
Sep 15 21:32:02.456417 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:02.456375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:32:02.456866 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:02.456476    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:32:02.469132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:02.469098    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:32:02.882882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:02.882800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:32:05.051566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.051497    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:32:05.143012 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.142976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m8s4\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy\") pod \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\" (UID: \"3eb78eb7-a2ec-4593-8f92-8b5cebf14618\") "
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:05.143485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143473    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:05.143800 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:32:05.143800 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143787    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.143880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143809    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istiod-ca-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.143880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143822    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-workload-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.143880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.143836    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-credential-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.144080 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.144036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:05.145699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.145671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:32:05.145699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.145695    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:05.145891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.145703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4" (OuterVolumeSpecName: "kube-api-access-6m8s4") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "kube-api-access-6m8s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:05.145891 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.145832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eb78eb7-a2ec-4593-8f92-8b5cebf14618" (UID: "3eb78eb7-a2ec-4593-8f92-8b5cebf14618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:05.244819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.244786    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-podinfo\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.244819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.244813    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-data\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.244819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.244821    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-token\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.245059 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.244831    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m8s4\" (UniqueName: \"kubernetes.io/projected/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-kube-api-access-6m8s4\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.245059 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.244841    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb78eb7-a2ec-4593-8f92-8b5cebf14618-istio-envoy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:05.889988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.889953    2578 generic.go:356] "Generic (PLEG): container finished" podID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerID="6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2" exitCode=0
Sep 15 21:32:05.890292 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.890006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"
Sep 15 21:32:05.890292 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.890040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerDied","Data":"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"}
Sep 15 21:32:05.890292 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.890106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9" event={"ID":"3eb78eb7-a2ec-4593-8f92-8b5cebf14618","Type":"ContainerDied","Data":"d3b5d9e32c64d9d3172ed4ec15e5b1608512034361ec08b2103b000a32e5ced0"}
Sep 15 21:32:05.890292 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.890138    2578 scope.go:117] "RemoveContainer" containerID="6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"
Sep 15 21:32:05.900296 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.900279    2578 scope.go:117] "RemoveContainer" containerID="4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"
Sep 15 21:32:05.909928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.909213    2578 scope.go:117] "RemoveContainer" containerID="6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"
Sep 15 21:32:05.909928 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:05.909543    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2\": container with ID starting with 6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2 not found: ID does not exist" containerID="6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"
Sep 15 21:32:05.909928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.909578    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2"} err="failed to get container status \"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2\": rpc error: code = NotFound desc = could not find container \"6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2\": container with ID starting with 6299fc6af508d4e192a8889950b0339ec678ebc1bc8d5b2d27044ccefc423db2 not found: ID does not exist"
Sep 15 21:32:05.909928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.909604    2578 scope.go:117] "RemoveContainer" containerID="4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"
Sep 15 21:32:05.910162 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:05.909964    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5\": container with ID starting with 4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5 not found: ID does not exist" containerID="4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"
Sep 15 21:32:05.910162 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.909995    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5"} err="failed to get container status \"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5\": rpc error: code = NotFound desc = could not find container \"4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5\": container with ID starting with 4b252a5c9d3205ba30970817dfec69282c056fc3dc243920b692cb0745caf4c5 not found: ID does not exist"
Sep 15 21:32:05.911705 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.911684    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:32:05.917229 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:05.917208    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d578a96c-openshift-default-5cf7fbbdd4-qrpm9"]
Sep 15 21:32:06.079657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.079624    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" path="/var/lib/kubelet/pods/3eb78eb7-a2ec-4593-8f92-8b5cebf14618/volumes"
Sep 15 21:32:06.335255 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.335220    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:32:06.335438 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.335406    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" podUID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" containerName="maas-api" containerID="cri-o://d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8" gracePeriod=30
Sep 15 21:32:06.496354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.496319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:32:06.496820 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.496774    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" podUID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" containerName="payload-processing" containerID="cri-o://12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746" gracePeriod=30
Sep 15 21:32:06.628346 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.627466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:32:06.760121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgvxp\" (UniqueName: \"kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp\") pod \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") "
Sep 15 21:32:06.760269 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") pod \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") "
Sep 15 21:32:06.760337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca\") pod \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") "
Sep 15 21:32:06.760337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs\") pod \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\" (UID: \"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1\") "
Sep 15 21:32:06.760737 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" (UID: "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:32:06.761001 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.760976    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:06.762599 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.762576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" (UID: "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:32:06.762703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.762618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" (UID: "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:32:06.762809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.762791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp" (OuterVolumeSpecName: "kube-api-access-lgvxp") pod "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" (UID: "8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1"). InnerVolumeSpecName "kube-api-access-lgvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:06.776482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.776454    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:32:06.861513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume\") pod \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") "
Sep 15 21:32:06.861513 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsdtj\" (UniqueName: \"kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj\") pod \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\" (UID: \"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207\") "
Sep 15 21:32:06.861772 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861758    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:06.861817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861774    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgvxp\" (UniqueName: \"kubernetes.io/projected/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-kube-api-access-lgvxp\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:06.861817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861786    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:06.861880 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.861835    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" (UID: "f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:32:06.863546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.863524    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj" (OuterVolumeSpecName: "kube-api-access-zsdtj") pod "f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" (UID: "f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207"). InnerVolumeSpecName "kube-api-access-zsdtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:06.896789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.896752    2578 generic.go:356] "Generic (PLEG): container finished" podID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" containerID="d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8" exitCode=0
Sep 15 21:32:06.896992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.896817    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"
Sep 15 21:32:06.896992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.896833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" event={"ID":"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1","Type":"ContainerDied","Data":"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"}
Sep 15 21:32:06.896992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.896937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj" event={"ID":"8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1","Type":"ContainerDied","Data":"dce28ab09e2b3ff3d3f83f45fe9166b56709d7a82f2606e6f32d3ca59f789513"}
Sep 15 21:32:06.896992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.896969    2578 scope.go:117] "RemoveContainer" containerID="d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"
Sep 15 21:32:06.898503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.898333    2578 generic.go:356] "Generic (PLEG): container finished" podID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" containerID="12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746" exitCode=0
Sep 15 21:32:06.898503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.898394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"
Sep 15 21:32:06.898503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.898402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" event={"ID":"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207","Type":"ContainerDied","Data":"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"}
Sep 15 21:32:06.898503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.898445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95" event={"ID":"f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207","Type":"ContainerDied","Data":"6e1b2fffacb31fea9ea3d8deb64bacc88d02a2f7d134c80514406f50fccf5823"}
Sep 15 21:32:06.907366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.907346    2578 scope.go:117] "RemoveContainer" containerID="d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"
Sep 15 21:32:06.907664 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:06.907641    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8\": container with ID starting with d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8 not found: ID does not exist" containerID="d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"
Sep 15 21:32:06.907793 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.907670    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8"} err="failed to get container status \"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8\": rpc error: code = NotFound desc = could not find container \"d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8\": container with ID starting with d2427fc70fa5614ff58d2f63a5502af0bd5cda3e0f66fee796a99226134f52c8 not found: ID does not exist"
Sep 15 21:32:06.907793 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.907688    2578 scope.go:117] "RemoveContainer" containerID="12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"
Sep 15 21:32:06.917060 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.917041    2578 scope.go:117] "RemoveContainer" containerID="12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"
Sep 15 21:32:06.917376 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:06.917356    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746\": container with ID starting with 12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746 not found: ID does not exist" containerID="12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"
Sep 15 21:32:06.917420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.917384    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746"} err="failed to get container status \"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746\": rpc error: code = NotFound desc = could not find container \"12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746\": container with ID starting with 12ff209d2da26e6f5cafbad0077c935aef48d00428f74a31dd8d67a78bea5746 not found: ID does not exist"
Sep 15 21:32:06.921906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.921881    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:32:06.926350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.926327    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-10c104-6b6c8d7744-778mj"]
Sep 15 21:32:06.935803 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.935774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:32:06.940676 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.940654    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-10c104-7b6556b647-rnc95"]
Sep 15 21:32:06.963273 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.963241    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdtj\" (UniqueName: \"kubernetes.io/projected/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-kube-api-access-zsdtj\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:06.963273 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:06.963266    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:08.075977 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:08.075945    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" path="/var/lib/kubelet/pods/8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1/volumes"
Sep 15 21:32:08.076357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:08.076287    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" path="/var/lib/kubelet/pods/f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207/volumes"
Sep 15 21:32:49.378954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:49.378882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:32:49.379876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:49.379102    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy" containerID="cri-o://6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2" gracePeriod=30
Sep 15 21:32:54.639196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.639127    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:32:54.743100 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743100 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2tlx\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743194    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743471    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket\") pod \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\" (UID: \"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0\") "
Sep 15 21:32:54.743615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:54.743615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:54.743878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:54.743878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:32:54.743988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:54.744035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.743996    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-credential-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.744035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.744016    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-data\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.744035 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.744032    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.744150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.744046    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-workload-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.744150 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.744061    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istiod-ca-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.745553 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.745525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx" (OuterVolumeSpecName: "kube-api-access-f2tlx") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "kube-api-access-f2tlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:54.745671 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.745610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:32:54.745983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.745965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:32:54.746047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.745985    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" (UID: "b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:32:54.845058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.845019    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-token\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.845058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.845055    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-envoy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.845268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.845070    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-istio-podinfo\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:54.845268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:54.845085    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tlx\" (UniqueName: \"kubernetes.io/projected/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0-kube-api-access-f2tlx\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:32:55.161835 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.161796    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerID="6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2" exitCode=0
Sep 15 21:32:55.162042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.161851    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"
Sep 15 21:32:55.162042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.161879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerDied","Data":"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"}
Sep 15 21:32:55.162042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.161950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2" event={"ID":"b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0","Type":"ContainerDied","Data":"0fa7b137e27d5c2c0f4ef93c900f52d9e5e6851ad651099d0af5fb2cbfdbca2b"}
Sep 15 21:32:55.162042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.161980    2578 scope.go:117] "RemoveContainer" containerID="6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"
Sep 15 21:32:55.171923 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.171903    2578 scope.go:117] "RemoveContainer" containerID="5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"
Sep 15 21:32:55.180176 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.180156    2578 scope.go:117] "RemoveContainer" containerID="6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"
Sep 15 21:32:55.180465 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:55.180439    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2\": container with ID starting with 6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2 not found: ID does not exist" containerID="6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"
Sep 15 21:32:55.180564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.180476    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2"} err="failed to get container status \"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2\": rpc error: code = NotFound desc = could not find container \"6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2\": container with ID starting with 6f8156d47c4968cb3ad2f9cfa24efbd0b9ddbdb1d7c125bfae097f3926ca9ac2 not found: ID does not exist"
Sep 15 21:32:55.180564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.180502    2578 scope.go:117] "RemoveContainer" containerID="5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"
Sep 15 21:32:55.180793 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:55.180767    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577\": container with ID starting with 5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577 not found: ID does not exist" containerID="5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"
Sep 15 21:32:55.180866 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.180797    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577"} err="failed to get container status \"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577\": rpc error: code = NotFound desc = could not find container \"5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577\": container with ID starting with 5bb316ba700c9c49e56ee672913cd7cf6b4e47914db0fac4ecc61016bf13b577 not found: ID does not exist"
Sep 15 21:32:55.186665 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.186638    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:32:55.192138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:55.192116    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-10c104-openshift-default-58f575cdcb-gppd2"]
Sep 15 21:32:56.076671 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.076641    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" path="/var/lib/kubelet/pods/b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0/volumes"
Sep 15 21:32:56.573596 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.573568    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:32:56.574119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574097    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574120    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574137    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" containerName="payload-processing"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574146    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" containerName="payload-processing"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574165    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574171    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574189    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" containerName="maas-api"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574198    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" containerName="maas-api"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574211    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.574234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574220    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574343    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574353    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8723ac7c-edf4-4eb7-8d4a-0a6f28c3b3c1" containerName="maas-api"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574360    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f8bfd7-6ba0-4719-9b2b-ffcf28a46207" containerName="payload-processing"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574378    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.574528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.574385    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb78eb7-a2ec-4593-8f92-8b5cebf14618" containerName="istio-proxy"
Sep 15 21:32:56.579149 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.579121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.580884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.580866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-32270d-serving-cert\""
Sep 15 21:32:56.589294 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.589269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:32:56.663264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.663228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.663469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.663304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.663469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.663336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.663558 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.663489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4nhf\" (UniqueName: \"kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.764351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.764298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.764351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.764358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.764595 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.764495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4nhf\" (UniqueName: \"kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.764595 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.764541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.764756 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:56.764706    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-serving-cert: secret "maas-api-e2e-explicit-32270d-serving-cert" not found
Sep 15 21:32:56.764822 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:32:56.764815    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls podName:42870b31-8949-4e08-8058-d1b6546c32cd nodeName:}" failed. No retries permitted until 2026-09-15 21:32:57.264794893 +0000 UTC m=+1005.665768136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls") pod "maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" (UID: "42870b31-8949-4e08-8058-d1b6546c32cd") : secret "maas-api-e2e-explicit-32270d-serving-cert" not found
Sep 15 21:32:56.765159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.765139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.766943 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.766924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:56.773132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:56.773104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4nhf\" (UniqueName: \"kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:57.255905 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.255862    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:32:57.256375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.256362    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:57.256426 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.256376    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d099ef-2cd3-4d31-a5f0-6868aa94c0f0" containerName="istio-proxy"
Sep 15 21:32:57.259860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.259839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.261781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.261760    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-32270d-dockercfg-6v78t\""
Sep 15 21:32:57.262332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.262316    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-32270d\""
Sep 15 21:32:57.270488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.270467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:57.272809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.272781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") pod \"maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:57.281081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.281008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:32:57.306493 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.306447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:32:57.310202 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.310174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.325068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.325030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:32:57.371809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.371776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rlr\" (UniqueName: \"kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.371980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.371916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.371980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.371962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbg9\" (UniqueName: \"kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.372055 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.372011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.473306 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.473265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.473515 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.473362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rlr\" (UniqueName: \"kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.473515 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.473457    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.473515 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.473503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbg9\" (UniqueName: \"kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.474190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.474166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.474268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.474222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.485077 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.485048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbg9\" (UniqueName: \"kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9\") pod \"payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.485197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.485055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rlr\" (UniqueName: \"kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr\") pod \"payload-processing-e2e-explicit-32270d-7b689f7b85-frj96\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") " pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.491729 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.491699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:57.571224 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.571189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:57.620876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.620838    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:57.785613 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.785588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:32:57.787587 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:32:57.787560    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42870b31_8949_4e08_8058_d1b6546c32cd.slice/crio-90a25993129894354611bd08fe5e7791916a9a45197a52ae98a986e8c96e7d15 WatchSource:0}: Error finding container 90a25993129894354611bd08fe5e7791916a9a45197a52ae98a986e8c96e7d15: Status 404 returned error can't find the container with id 90a25993129894354611bd08fe5e7791916a9a45197a52ae98a986e8c96e7d15
Sep 15 21:32:57.788877 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.788857    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:32:57.874993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.874866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:32:57.877997 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:32:57.877965    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ede9fef_bd2a_46ed_a2ea_31c73fcd77f4.slice/crio-0d1da86cb0170575102a18808fa3f9c35e83e42acdb2b83548867871e127c966 WatchSource:0}: Error finding container 0d1da86cb0170575102a18808fa3f9c35e83e42acdb2b83548867871e127c966: Status 404 returned error can't find the container with id 0d1da86cb0170575102a18808fa3f9c35e83e42acdb2b83548867871e127c966
Sep 15 21:32:57.919284 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:57.919236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:32:57.923569 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:32:57.923542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9883f731_202c_47f4_8f95_b4a633d5f9db.slice/crio-3e01c96fdb0cf377ed93943001452af04333c373c45683e2d01fe6197c93fe5d WatchSource:0}: Error finding container 3e01c96fdb0cf377ed93943001452af04333c373c45683e2d01fe6197c93fe5d: Status 404 returned error can't find the container with id 3e01c96fdb0cf377ed93943001452af04333c373c45683e2d01fe6197c93fe5d
Sep 15 21:32:58.182712 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.182673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" event={"ID":"9883f731-202c-47f4-8f95-b4a633d5f9db","Type":"ContainerStarted","Data":"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"}
Sep 15 21:32:58.182878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.182756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" event={"ID":"9883f731-202c-47f4-8f95-b4a633d5f9db","Type":"ContainerStarted","Data":"3e01c96fdb0cf377ed93943001452af04333c373c45683e2d01fe6197c93fe5d"}
Sep 15 21:32:58.182957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.182921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:32:58.184755 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.184707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" event={"ID":"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4","Type":"ContainerStarted","Data":"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"}
Sep 15 21:32:58.184965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.184948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:32:58.185143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.185096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" event={"ID":"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4","Type":"ContainerStarted","Data":"0d1da86cb0170575102a18808fa3f9c35e83e42acdb2b83548867871e127c966"}
Sep 15 21:32:58.186261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.186236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" event={"ID":"42870b31-8949-4e08-8058-d1b6546c32cd","Type":"ContainerStarted","Data":"90a25993129894354611bd08fe5e7791916a9a45197a52ae98a986e8c96e7d15"}
Sep 15 21:32:58.205815 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.205752    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" podStartSLOduration=1.205729773 podStartE2EDuration="1.205729773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:58.201926714 +0000 UTC m=+1006.602899962" watchObservedRunningTime="2026-09-15 21:32:58.205729773 +0000 UTC m=+1006.606703014"
Sep 15 21:32:58.234452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:58.234388    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" podStartSLOduration=1.234364408 podStartE2EDuration="1.234364408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:58.232543406 +0000 UTC m=+1006.633516652" watchObservedRunningTime="2026-09-15 21:32:58.234364408 +0000 UTC m=+1006.635337656"
Sep 15 21:32:59.193258 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:59.193226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" event={"ID":"42870b31-8949-4e08-8058-d1b6546c32cd","Type":"ContainerStarted","Data":"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"}
Sep 15 21:32:59.193657 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:59.193361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:32:59.213624 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:32:59.213574    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" podStartSLOduration=2.861710003 podStartE2EDuration="3.213555338s" totalImagesPullingTime="351.845335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:59.211881711 +0000 UTC m=+1007.612854957" watchObservedRunningTime="2026-09-15 21:32:59.213555338 +0000 UTC m=+1007.614528584"
Sep 15 21:33:05.204546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:05.204510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:33:09.195262 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:09.195228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:33:10.200559 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:10.200526    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:33:45.832497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.832458    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:33:45.836643 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.836623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:45.838449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.838425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1664eb4-serving-cert\""
Sep 15 21:33:45.846037 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.846013    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:33:45.977321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.977274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:45.977531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.977343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hv7\" (UniqueName: \"kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:45.977531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.977494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:45.977645 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:45.977528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.078710 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.078672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.078925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.078747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.078925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.078802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.078925 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:33:46.078818    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-serving-cert: secret "maas-api-e2e-mt-e1664eb4-serving-cert" not found
Sep 15 21:33:46.078925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.078845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hv7\" (UniqueName: \"kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.078925 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:33:46.078890    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls podName:80218b4e-ec48-4a3b-a3af-9306eb4404e0 nodeName:}" failed. No retries permitted until 2026-09-15 21:33:46.578872663 +0000 UTC m=+1054.979845889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls") pod "maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" (UID: "80218b4e-ec48-4a3b-a3af-9306eb4404e0") : secret "maas-api-e2e-mt-e1664eb4-serving-cert" not found
Sep 15 21:33:46.079372 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.079353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.081240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.081216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.087785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.087713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hv7\" (UniqueName: \"kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.584711 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.584670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.587128 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.587102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") pod \"maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.683139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.683097    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:33:46.687349 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.687324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.689264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.689239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1664eb4-dockercfg-682b5\""
Sep 15 21:33:46.689406 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.689241    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1664eb4\""
Sep 15 21:33:46.697637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.697607    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:33:46.750180 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.750131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:46.786469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.786434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.786761 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.786729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqw5\" (UniqueName: \"kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.888032 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.887949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqw5\" (UniqueName: \"kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.888032 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.888008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.888646 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.888625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.895529 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.895506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqw5\" (UniqueName: \"kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5\") pod \"payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") " pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:46.999519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:46.999486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:47.028471 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.028446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:33:47.030602 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:33:47.030561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80218b4e_ec48_4a3b_a3af_9306eb4404e0.slice/crio-45dd8dfaf440232b397aa00a88374d6acffb5aa89e00c3a2e4e327fa582edeca WatchSource:0}: Error finding container 45dd8dfaf440232b397aa00a88374d6acffb5aa89e00c3a2e4e327fa582edeca: Status 404 returned error can't find the container with id 45dd8dfaf440232b397aa00a88374d6acffb5aa89e00c3a2e4e327fa582edeca
Sep 15 21:33:47.282606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.282579    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:33:47.285181 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:33:47.285151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69532069_ac42_4047_8f66_846d44a3aa0f.slice/crio-45a0f509b9f7e9d65e6b46dfce5a069ca4593becfe2e3be9b73d3ed1f0dea513 WatchSource:0}: Error finding container 45a0f509b9f7e9d65e6b46dfce5a069ca4593becfe2e3be9b73d3ed1f0dea513: Status 404 returned error can't find the container with id 45a0f509b9f7e9d65e6b46dfce5a069ca4593becfe2e3be9b73d3ed1f0dea513
Sep 15 21:33:47.454264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.454223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" event={"ID":"69532069-ac42-4047-8f66-846d44a3aa0f","Type":"ContainerStarted","Data":"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"}
Sep 15 21:33:47.454452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.454298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:33:47.454452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.454325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" event={"ID":"69532069-ac42-4047-8f66-846d44a3aa0f","Type":"ContainerStarted","Data":"45a0f509b9f7e9d65e6b46dfce5a069ca4593becfe2e3be9b73d3ed1f0dea513"}
Sep 15 21:33:47.455628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.455596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" event={"ID":"80218b4e-ec48-4a3b-a3af-9306eb4404e0","Type":"ContainerStarted","Data":"45dd8dfaf440232b397aa00a88374d6acffb5aa89e00c3a2e4e327fa582edeca"}
Sep 15 21:33:47.471197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:47.471140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" podStartSLOduration=1.47112483 podStartE2EDuration="1.47112483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:47.469370181 +0000 UTC m=+1055.870343428" watchObservedRunningTime="2026-09-15 21:33:47.47112483 +0000 UTC m=+1055.872098075"
Sep 15 21:33:48.462647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:48.462606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" event={"ID":"80218b4e-ec48-4a3b-a3af-9306eb4404e0","Type":"ContainerStarted","Data":"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"}
Sep 15 21:33:48.463085 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:48.462699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:48.481577 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:48.481526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" podStartSLOduration=2.926370087 podStartE2EDuration="3.481508908s" totalImagesPullingTime="555.138821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:48.480896552 +0000 UTC m=+1056.881869803" watchObservedRunningTime="2026-09-15 21:33:48.481508908 +0000 UTC m=+1056.882482199"
Sep 15 21:33:54.473223 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:54.473190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:33:58.464897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:33:58.464862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:34:30.183485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.183451    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:34:30.184140 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.183648    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" podUID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" containerName="maas-api" containerID="cri-o://2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3" gracePeriod=30
Sep 15 21:34:30.247470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.247433    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:34:30.247732 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.247691    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" podUID="69532069-ac42-4047-8f66-846d44a3aa0f" containerName="payload-processing" containerID="cri-o://8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28" gracePeriod=30
Sep 15 21:34:30.529089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.528786    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:34:30.545484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.545459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8hv7\" (UniqueName: \"kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7\") pod \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") "
Sep 15 21:34:30.545602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.545514    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs\") pod \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") "
Sep 15 21:34:30.545602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.545561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") pod \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") "
Sep 15 21:34:30.545686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.545633    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca\") pod \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\" (UID: \"80218b4e-ec48-4a3b-a3af-9306eb4404e0\") "
Sep 15 21:34:30.546080 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.546049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80218b4e-ec48-4a3b-a3af-9306eb4404e0" (UID: "80218b4e-ec48-4a3b-a3af-9306eb4404e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:34:30.547616 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.547594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7" (OuterVolumeSpecName: "kube-api-access-m8hv7") pod "80218b4e-ec48-4a3b-a3af-9306eb4404e0" (UID: "80218b4e-ec48-4a3b-a3af-9306eb4404e0"). InnerVolumeSpecName "kube-api-access-m8hv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:30.547857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.547839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80218b4e-ec48-4a3b-a3af-9306eb4404e0" (UID: "80218b4e-ec48-4a3b-a3af-9306eb4404e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:30.548020 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.547997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80218b4e-ec48-4a3b-a3af-9306eb4404e0" (UID: "80218b4e-ec48-4a3b-a3af-9306eb4404e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:34:30.556882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.556853    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:34:30.646695 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.646659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume\") pod \"69532069-ac42-4047-8f66-846d44a3aa0f\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") "
Sep 15 21:34:30.646886 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.646790    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lqw5\" (UniqueName: \"kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5\") pod \"69532069-ac42-4047-8f66-846d44a3aa0f\" (UID: \"69532069-ac42-4047-8f66-846d44a3aa0f\") "
Sep 15 21:34:30.647048 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.647024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69532069-ac42-4047-8f66-846d44a3aa0f" (UID: "69532069-ac42-4047-8f66-846d44a3aa0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:34:30.647143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.647044    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hv7\" (UniqueName: \"kubernetes.io/projected/80218b4e-ec48-4a3b-a3af-9306eb4404e0-kube-api-access-m8hv7\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:30.647143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.647063    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:30.647143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.647074    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80218b4e-ec48-4a3b-a3af-9306eb4404e0-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:30.647143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.647081    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80218b4e-ec48-4a3b-a3af-9306eb4404e0-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:30.648817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.648794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5" (OuterVolumeSpecName: "kube-api-access-2lqw5") pod "69532069-ac42-4047-8f66-846d44a3aa0f" (UID: "69532069-ac42-4047-8f66-846d44a3aa0f"). InnerVolumeSpecName "kube-api-access-2lqw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:30.694516 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.694479    2578 generic.go:356] "Generic (PLEG): container finished" podID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" containerID="2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3" exitCode=0
Sep 15 21:34:30.694701 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.694569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" event={"ID":"80218b4e-ec48-4a3b-a3af-9306eb4404e0","Type":"ContainerDied","Data":"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"}
Sep 15 21:34:30.694701 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.694677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22" event={"ID":"80218b4e-ec48-4a3b-a3af-9306eb4404e0","Type":"ContainerDied","Data":"45dd8dfaf440232b397aa00a88374d6acffb5aa89e00c3a2e4e327fa582edeca"}
Sep 15 21:34:30.694843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.694582    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"
Sep 15 21:34:30.694843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.694711    2578 scope.go:117] "RemoveContainer" containerID="2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"
Sep 15 21:34:30.696242 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.696218    2578 generic.go:356] "Generic (PLEG): container finished" podID="69532069-ac42-4047-8f66-846d44a3aa0f" containerID="8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28" exitCode=0
Sep 15 21:34:30.696339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.696272    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"
Sep 15 21:34:30.696339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.696295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" event={"ID":"69532069-ac42-4047-8f66-846d44a3aa0f","Type":"ContainerDied","Data":"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"}
Sep 15 21:34:30.696339 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.696335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2" event={"ID":"69532069-ac42-4047-8f66-846d44a3aa0f","Type":"ContainerDied","Data":"45a0f509b9f7e9d65e6b46dfce5a069ca4593becfe2e3be9b73d3ed1f0dea513"}
Sep 15 21:34:30.706266 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.706240    2578 scope.go:117] "RemoveContainer" containerID="2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"
Sep 15 21:34:30.706875 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:34:30.706844    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3\": container with ID starting with 2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3 not found: ID does not exist" containerID="2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"
Sep 15 21:34:30.706988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.706884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3"} err="failed to get container status \"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3\": rpc error: code = NotFound desc = could not find container \"2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3\": container with ID starting with 2c6fe60bc3fa3df65a6c1a7346f2d1f023edc9bfdac1c17c7ebce1ad564054c3 not found: ID does not exist"
Sep 15 21:34:30.706988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.706903    2578 scope.go:117] "RemoveContainer" containerID="8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"
Sep 15 21:34:30.715570 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.715550    2578 scope.go:117] "RemoveContainer" containerID="8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"
Sep 15 21:34:30.715910 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:34:30.715888    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28\": container with ID starting with 8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28 not found: ID does not exist" containerID="8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"
Sep 15 21:34:30.715985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.715916    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28"} err="failed to get container status \"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28\": rpc error: code = NotFound desc = could not find container \"8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28\": container with ID starting with 8c2ad4b65f7057ff811e1036503b37870581b7748ff35a1dcf9f077116bfaf28 not found: ID does not exist"
Sep 15 21:34:30.722700 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.722678    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:34:30.729123 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.729072    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1664eb4-74f55699d6-4wx22"]
Sep 15 21:34:30.739647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.739628    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:34:30.743985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.743963    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1664eb4-6c49d56cc-mzcf2"]
Sep 15 21:34:30.748068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.748046    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqw5\" (UniqueName: \"kubernetes.io/projected/69532069-ac42-4047-8f66-846d44a3aa0f-kube-api-access-2lqw5\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:30.748068 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:30.748069    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69532069-ac42-4047-8f66-846d44a3aa0f-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:34:32.077043 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:32.077009    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69532069-ac42-4047-8f66-846d44a3aa0f" path="/var/lib/kubelet/pods/69532069-ac42-4047-8f66-846d44a3aa0f/volumes"
Sep 15 21:34:32.077426 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:32.077386    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" path="/var/lib/kubelet/pods/80218b4e-ec48-4a3b-a3af-9306eb4404e0/volumes"
Sep 15 21:34:40.323490 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.323449    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:34:40.324156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324133    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69532069-ac42-4047-8f66-846d44a3aa0f" containerName="payload-processing"
Sep 15 21:34:40.324253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69532069-ac42-4047-8f66-846d44a3aa0f" containerName="payload-processing"
Sep 15 21:34:40.324253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324172    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" containerName="maas-api"
Sep 15 21:34:40.324253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324180    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" containerName="maas-api"
Sep 15 21:34:40.324566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324279    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80218b4e-ec48-4a3b-a3af-9306eb4404e0" containerName="maas-api"
Sep 15 21:34:40.324566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.324296    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69532069-ac42-4047-8f66-846d44a3aa0f" containerName="payload-processing"
Sep 15 21:34:40.327849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.327828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.330222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.330198    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-32270d\"/\"kube-root-ca.crt\""
Sep 15 21:34:40.330324 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.330253    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-32270d\"/\"openshift-service-ca.crt\""
Sep 15 21:34:40.330324 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.330253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-32270d\"/\"explicit-model-32270d-kserve-self-signed-certs\""
Sep 15 21:34:40.330441 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.330200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-32270d\"/\"default-dockercfg-t9lc7\""
Sep 15 21:34:40.336693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.336668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:34:40.449874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.449821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.449874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.449881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.450143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.449941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.450143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.449974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwvl\" (UniqueName: \"kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.450143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.450040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.450143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.450131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.550913 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.550878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.550927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.550953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.550981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.551005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwvl\" (UniqueName: \"kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.551050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551338 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.551316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.551368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.551474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.551449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.553313 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.553291    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.553479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.553462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.558930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.558905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwvl\" (UniqueName: \"kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl\") pod \"explicit-model-32270d-kserve-744c769bb5-db2ds\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") " pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.639753 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.639646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:40.924874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:40.924758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:34:40.927876 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:34:40.927845    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c4131ba_c152_48f1_b848_95efd3408b1c.slice/crio-ca82d862a4de017d3065d4519f4b454b776cfd815adf24c6988e59d41522143b WatchSource:0}: Error finding container ca82d862a4de017d3065d4519f4b454b776cfd815adf24c6988e59d41522143b: Status 404 returned error can't find the container with id ca82d862a4de017d3065d4519f4b454b776cfd815adf24c6988e59d41522143b
Sep 15 21:34:41.759261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:41.759223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" event={"ID":"7c4131ba-c152-48f1-b848-95efd3408b1c","Type":"ContainerStarted","Data":"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"}
Sep 15 21:34:41.759261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:41.759267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" event={"ID":"7c4131ba-c152-48f1-b848-95efd3408b1c","Type":"ContainerStarted","Data":"ca82d862a4de017d3065d4519f4b454b776cfd815adf24c6988e59d41522143b"}
Sep 15 21:34:41.777845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:41.777790    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" podStartSLOduration=1.777771801 podStartE2EDuration="1.777771801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:41.775019563 +0000 UTC m=+1110.175992810" watchObservedRunningTime="2026-09-15 21:34:41.777771801 +0000 UTC m=+1110.178745108"
Sep 15 21:34:50.640248 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:50.640204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:50.640248 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:50.640258    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:50.653078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:50.653048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:34:50.819334 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:34:50.819309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:35:00.692419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.692380    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:35:00.696435 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.696418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.699441 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.699422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2d4fab-dockercfg-nv6mc\""
Sep 15 21:35:00.700006 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.699981    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2d4fab\""
Sep 15 21:35:00.714641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.714614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:35:00.763174 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.763134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.763368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.763315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr94\" (UniqueName: \"kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.863924 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.863890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr94\" (UniqueName: \"kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.864119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.863949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.864650 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.864630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:00.887156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:00.887119    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr94\" (UniqueName: \"kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94\") pod \"payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:01.007354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.007269    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:01.283178 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.283149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:35:01.285065 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:35:01.285039    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e54cd9b_7a2e_43c4_b650_c5e7ae51a771.slice/crio-7fef5b864c50c27172f568b55283725bd58eb8df5730f21d1510900cd3ca880c WatchSource:0}: Error finding container 7fef5b864c50c27172f568b55283725bd58eb8df5730f21d1510900cd3ca880c: Status 404 returned error can't find the container with id 7fef5b864c50c27172f568b55283725bd58eb8df5730f21d1510900cd3ca880c
Sep 15 21:35:01.873994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.873962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" event={"ID":"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771","Type":"ContainerStarted","Data":"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"}
Sep 15 21:35:01.874414 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.874021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" event={"ID":"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771","Type":"ContainerStarted","Data":"7fef5b864c50c27172f568b55283725bd58eb8df5730f21d1510900cd3ca880c"}
Sep 15 21:35:01.874414 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.874127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:01.901972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:01.901915    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" podStartSLOduration=1.901895763 podStartE2EDuration="1.901895763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:01.899459337 +0000 UTC m=+1130.300432584" watchObservedRunningTime="2026-09-15 21:35:01.901895763 +0000 UTC m=+1130.302869054"
Sep 15 21:35:12.882363 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:12.882321    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:35:24.414504 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.414465    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:35:24.420691 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.420658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.422683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.422657    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"openshift-service-ca.crt\""
Sep 15 21:35:24.422828 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.422773    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"default-dockercfg-brd6c\""
Sep 15 21:35:24.423021 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.422983    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"e2e-conflict-a-473af5c1-kserve-self-signed-certs\""
Sep 15 21:35:24.423129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.423069    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac4411\"/\"kube-root-ca.crt\""
Sep 15 21:35:24.430785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.429167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:35:24.498282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.498470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.498470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.498470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv5s\" (UniqueName: \"kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.498470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.498470 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.498462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599442 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv5s\" (UniqueName: \"kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.599946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.600001 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.599929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.600119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.600093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.601936 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.601914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.602029 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.601982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.606471 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.606447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv5s\" (UniqueName: \"kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s\") pod \"e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") " pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:24.733642 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:24.733601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:25.010633 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:25.010551    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:35:25.013909 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:35:25.013881    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1859a80_4fe5_4230_a057_bf74f4c56af6.slice/crio-2d73fde4b9f2ec92f960a2cfcecf513fa043a322dad6095816dbe7c008eb8150 WatchSource:0}: Error finding container 2d73fde4b9f2ec92f960a2cfcecf513fa043a322dad6095816dbe7c008eb8150: Status 404 returned error can't find the container with id 2d73fde4b9f2ec92f960a2cfcecf513fa043a322dad6095816dbe7c008eb8150
Sep 15 21:35:26.019197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:26.019160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" event={"ID":"b1859a80-4fe5-4230-a057-bf74f4c56af6","Type":"ContainerStarted","Data":"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"}
Sep 15 21:35:26.019197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:26.019206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" event={"ID":"b1859a80-4fe5-4230-a057-bf74f4c56af6","Type":"ContainerStarted","Data":"2d73fde4b9f2ec92f960a2cfcecf513fa043a322dad6095816dbe7c008eb8150"}
Sep 15 21:35:26.044736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:26.043271    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" podStartSLOduration=2.043242104 podStartE2EDuration="2.043242104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:26.039632849 +0000 UTC m=+1154.440606096" watchObservedRunningTime="2026-09-15 21:35:26.043242104 +0000 UTC m=+1154.444215351"
Sep 15 21:35:27.411762 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.410575    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:35:27.411762 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.410845    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" podUID="42870b31-8949-4e08-8058-d1b6546c32cd" containerName="maas-api" containerID="cri-o://3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da" gracePeriod=30
Sep 15 21:35:27.479198 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.479160    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:35:27.479645 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.479609    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" podUID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" containerName="payload-processing" containerID="cri-o://3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9" gracePeriod=30
Sep 15 21:35:27.489459 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.489429    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:35:27.490353 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.489659    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" podUID="9883f731-202c-47f4-8f95-b4a633d5f9db" containerName="payload-pre-processing" containerID="cri-o://eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064" gracePeriod=30
Sep 15 21:35:27.767792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.767763    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:35:27.791042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.791017    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:35:27.793528 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.793509    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:35:27.835072 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.835036    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") pod \"42870b31-8949-4e08-8058-d1b6546c32cd\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") "
Sep 15 21:35:27.835257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.835104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca\") pod \"42870b31-8949-4e08-8058-d1b6546c32cd\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") "
Sep 15 21:35:27.835257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.835187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4nhf\" (UniqueName: \"kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf\") pod \"42870b31-8949-4e08-8058-d1b6546c32cd\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") "
Sep 15 21:35:27.835386 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.835358    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs\") pod \"42870b31-8949-4e08-8058-d1b6546c32cd\" (UID: \"42870b31-8949-4e08-8058-d1b6546c32cd\") "
Sep 15 21:35:27.835541 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.835516    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42870b31-8949-4e08-8058-d1b6546c32cd" (UID: "42870b31-8949-4e08-8058-d1b6546c32cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:35:27.836406 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.836354    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42870b31-8949-4e08-8058-d1b6546c32cd-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:27.837975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.837952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42870b31-8949-4e08-8058-d1b6546c32cd" (UID: "42870b31-8949-4e08-8058-d1b6546c32cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:35:27.838108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.838085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf" (OuterVolumeSpecName: "kube-api-access-f4nhf") pod "42870b31-8949-4e08-8058-d1b6546c32cd" (UID: "42870b31-8949-4e08-8058-d1b6546c32cd"). InnerVolumeSpecName "kube-api-access-f4nhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:27.839389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.839360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42870b31-8949-4e08-8058-d1b6546c32cd" (UID: "42870b31-8949-4e08-8058-d1b6546c32cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:35:27.937410 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.937380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume\") pod \"9883f731-202c-47f4-8f95-b4a633d5f9db\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") "
Sep 15 21:35:27.937588 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.937450    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxbg9\" (UniqueName: \"kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9\") pod \"9883f731-202c-47f4-8f95-b4a633d5f9db\" (UID: \"9883f731-202c-47f4-8f95-b4a633d5f9db\") "
Sep 15 21:35:27.937588 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.937538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume\") pod \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") "
Sep 15 21:35:27.937588 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.937580    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86rlr\" (UniqueName: \"kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr\") pod \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\" (UID: \"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4\") "
Sep 15 21:35:27.937863 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.937833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9883f731-202c-47f4-8f95-b4a633d5f9db" (UID: "9883f731-202c-47f4-8f95-b4a633d5f9db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:35:27.938282 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.938256    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4nhf\" (UniqueName: \"kubernetes.io/projected/42870b31-8949-4e08-8058-d1b6546c32cd-kube-api-access-f4nhf\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:27.938416 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.938282    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:27.938416 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.938296    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9883f731-202c-47f4-8f95-b4a633d5f9db-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:27.938416 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.938295    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" (UID: "0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:35:27.938416 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.938310    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42870b31-8949-4e08-8058-d1b6546c32cd-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:27.940055 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.940024    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr" (OuterVolumeSpecName: "kube-api-access-86rlr") pod "0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" (UID: "0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4"). InnerVolumeSpecName "kube-api-access-86rlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:27.940163 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:27.940139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9" (OuterVolumeSpecName: "kube-api-access-wxbg9") pod "9883f731-202c-47f4-8f95-b4a633d5f9db" (UID: "9883f731-202c-47f4-8f95-b4a633d5f9db"). InnerVolumeSpecName "kube-api-access-wxbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:28.032492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.032282    2578 generic.go:356] "Generic (PLEG): container finished" podID="42870b31-8949-4e08-8058-d1b6546c32cd" containerID="3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da" exitCode=0
Sep 15 21:35:28.032492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.032372    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" event={"ID":"42870b31-8949-4e08-8058-d1b6546c32cd","Type":"ContainerDied","Data":"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"}
Sep 15 21:35:28.032492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.032453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt" event={"ID":"42870b31-8949-4e08-8058-d1b6546c32cd","Type":"ContainerDied","Data":"90a25993129894354611bd08fe5e7791916a9a45197a52ae98a986e8c96e7d15"}
Sep 15 21:35:28.032492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.032471    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"
Sep 15 21:35:28.032492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.032486    2578 scope.go:117] "RemoveContainer" containerID="3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"
Sep 15 21:35:28.034110 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.034084    2578 generic.go:356] "Generic (PLEG): container finished" podID="9883f731-202c-47f4-8f95-b4a633d5f9db" containerID="eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064" exitCode=0
Sep 15 21:35:28.034247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.034143    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"
Sep 15 21:35:28.034247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.034154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" event={"ID":"9883f731-202c-47f4-8f95-b4a633d5f9db","Type":"ContainerDied","Data":"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"}
Sep 15 21:35:28.034247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.034209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2" event={"ID":"9883f731-202c-47f4-8f95-b4a633d5f9db","Type":"ContainerDied","Data":"3e01c96fdb0cf377ed93943001452af04333c373c45683e2d01fe6197c93fe5d"}
Sep 15 21:35:28.035454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.035429    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" containerID="3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9" exitCode=0
Sep 15 21:35:28.035580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.035470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"
Sep 15 21:35:28.035580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.035506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" event={"ID":"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4","Type":"ContainerDied","Data":"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"}
Sep 15 21:35:28.035580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.035562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96" event={"ID":"0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4","Type":"ContainerDied","Data":"0d1da86cb0170575102a18808fa3f9c35e83e42acdb2b83548867871e127c966"}
Sep 15 21:35:28.038869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.038849    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:28.038962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.038873    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rlr\" (UniqueName: \"kubernetes.io/projected/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4-kube-api-access-86rlr\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:28.038962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.038887    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbg9\" (UniqueName: \"kubernetes.io/projected/9883f731-202c-47f4-8f95-b4a633d5f9db-kube-api-access-wxbg9\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:28.044287 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.044266    2578 scope.go:117] "RemoveContainer" containerID="3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"
Sep 15 21:35:28.044611 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:28.044589    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da\": container with ID starting with 3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da not found: ID does not exist" containerID="3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"
Sep 15 21:35:28.044703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.044622    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da"} err="failed to get container status \"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da\": rpc error: code = NotFound desc = could not find container \"3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da\": container with ID starting with 3f9b8a4157628e880f41a4f158fdfe262da0923bd309223957475383a75a55da not found: ID does not exist"
Sep 15 21:35:28.044703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.044643    2578 scope.go:117] "RemoveContainer" containerID="eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"
Sep 15 21:35:28.053091 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.053071    2578 scope.go:117] "RemoveContainer" containerID="eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"
Sep 15 21:35:28.053353 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:28.053334    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064\": container with ID starting with eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064 not found: ID does not exist" containerID="eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"
Sep 15 21:35:28.053408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.053360    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064"} err="failed to get container status \"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064\": rpc error: code = NotFound desc = could not find container \"eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064\": container with ID starting with eefa1118116d7a25ca4285d31561731f60b7fb8dcdefe43ca5c8bd19740ce064 not found: ID does not exist"
Sep 15 21:35:28.053408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.053377    2578 scope.go:117] "RemoveContainer" containerID="3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"
Sep 15 21:35:28.063945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.062449    2578 scope.go:117] "RemoveContainer" containerID="3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"
Sep 15 21:35:28.064137 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:28.064117    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9\": container with ID starting with 3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9 not found: ID does not exist" containerID="3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"
Sep 15 21:35:28.064197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.064148    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9"} err="failed to get container status \"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9\": rpc error: code = NotFound desc = could not find container \"3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9\": container with ID starting with 3ca7782c6c13a84392904ae3724c941c6911b6698c0756f74834c1160114c6c9 not found: ID does not exist"
Sep 15 21:35:28.069196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.069169    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:35:28.078425 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.078401    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-32270d-6d7fb98cbd-shtgt"]
Sep 15 21:35:28.084505 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.084472    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:35:28.089394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.089367    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-32270d-7b689f7b85-frj96"]
Sep 15 21:35:28.101571 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.101540    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:35:28.112798 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:28.112759    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-32270d-58b7c6cf5d-ktmr2"]
Sep 15 21:35:30.076252 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:30.076219    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" path="/var/lib/kubelet/pods/0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4/volumes"
Sep 15 21:35:30.076654 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:30.076630    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42870b31-8949-4e08-8058-d1b6546c32cd" path="/var/lib/kubelet/pods/42870b31-8949-4e08-8058-d1b6546c32cd/volumes"
Sep 15 21:35:30.076993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:30.076979    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9883f731-202c-47f4-8f95-b4a633d5f9db" path="/var/lib/kubelet/pods/9883f731-202c-47f4-8f95-b4a633d5f9db/volumes"
Sep 15 21:35:33.101568 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.101531    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:35:33.463922 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.463887    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:35:33.464609 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464585    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42870b31-8949-4e08-8058-d1b6546c32cd" containerName="maas-api"
Sep 15 21:35:33.464686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="42870b31-8949-4e08-8058-d1b6546c32cd" containerName="maas-api"
Sep 15 21:35:33.464686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464650    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" containerName="payload-processing"
Sep 15 21:35:33.464686 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464658    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" containerName="payload-processing"
Sep 15 21:35:33.464853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9883f731-202c-47f4-8f95-b4a633d5f9db" containerName="payload-pre-processing"
Sep 15 21:35:33.464853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464698    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9883f731-202c-47f4-8f95-b4a633d5f9db" containerName="payload-pre-processing"
Sep 15 21:35:33.464853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464816    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9883f731-202c-47f4-8f95-b4a633d5f9db" containerName="payload-pre-processing"
Sep 15 21:35:33.464853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464836    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="42870b31-8949-4e08-8058-d1b6546c32cd" containerName="maas-api"
Sep 15 21:35:33.464853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.464853    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ede9fef-bd2a-46ed-a2ea-31c73fcd77f4" containerName="payload-processing"
Sep 15 21:35:33.469847 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.469829    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.472649 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.472622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fb0f0e-serving-cert\""
Sep 15 21:35:33.479533 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.479496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:35:33.598315 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.598276    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.598519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.598361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.598519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.598422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.598519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.598486    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrz5\" (UniqueName: \"kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.699558 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.699511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrz5\" (UniqueName: \"kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.699819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.699680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.699819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.699801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.699947 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.699857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.699999 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:33.699946    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-serving-cert: secret "maas-api-e2e-shared-b-w2-fb0f0e-serving-cert" not found
Sep 15 21:35:33.700055 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:33.700042    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls podName:9233d557-a6aa-4d6a-aecd-58cab5f517cd nodeName:}" failed. No retries permitted until 2026-09-15 21:35:34.200021997 +0000 UTC m=+1162.600995234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" (UID: "9233d557-a6aa-4d6a-aecd-58cab5f517cd") : secret "maas-api-e2e-shared-b-w2-fb0f0e-serving-cert" not found
Sep 15 21:35:33.700526 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.700508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.702167 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.702145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:33.709162 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:33.709135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrz5\" (UniqueName: \"kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:34.205703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.205407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:34.208209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.208183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:34.382592 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.382555    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:34.448946 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.448912    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:35:34.455682 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.455662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.459118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.459058    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fb0f0e\""
Sep 15 21:35:34.459334 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.459318    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fb0f0e-dockercfg-wwxlt\""
Sep 15 21:35:34.470707 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.470680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:35:34.530958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.530922    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:35:34.534696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.534675    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.547841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.547811    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:35:34.610047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.610013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.610209 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.610092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnb9\" (UniqueName: \"kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.681239 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.681209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:35:34.683479 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:35:34.683453    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9233d557_a6aa_4d6a_aecd_58cab5f517cd.slice/crio-25ed5fca5355c1a2cd314beca77d35205ec3ad87d9b6a3b3889bc20e8294777a WatchSource:0}: Error finding container 25ed5fca5355c1a2cd314beca77d35205ec3ad87d9b6a3b3889bc20e8294777a: Status 404 returned error can't find the container with id 25ed5fca5355c1a2cd314beca77d35205ec3ad87d9b6a3b3889bc20e8294777a
Sep 15 21:35:34.711455 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.711391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.711455 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.711437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnb9\" (UniqueName: \"kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.711606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.711521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.711606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.711537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5bl\" (UniqueName: \"kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.712191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.712172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.722205 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.722183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnb9\" (UniqueName: \"kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9\") pod \"payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.734053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.733969    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:34.734160 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.734079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:34.746735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.746697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:34.770267 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.770235    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:34.813053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.813012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.813254 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.813176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5bl\" (UniqueName: \"kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.813847 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.813823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.824358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.824328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5bl\" (UniqueName: \"kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl\") pod \"payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:34.844839 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:34.844809    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:35.062181 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.062154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:35:35.064346 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:35:35.064308    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bd7051_8a23_4306_a4cf_40d95ac44f50.slice/crio-02a739cfeb0c89837b405cb788d3ee24f00eec70d2fc650cb24de52a5dbd7c28 WatchSource:0}: Error finding container 02a739cfeb0c89837b405cb788d3ee24f00eec70d2fc650cb24de52a5dbd7c28: Status 404 returned error can't find the container with id 02a739cfeb0c89837b405cb788d3ee24f00eec70d2fc650cb24de52a5dbd7c28
Sep 15 21:35:35.078978 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.078946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" event={"ID":"f5bd7051-8a23-4306-a4cf-40d95ac44f50","Type":"ContainerStarted","Data":"02a739cfeb0c89837b405cb788d3ee24f00eec70d2fc650cb24de52a5dbd7c28"}
Sep 15 21:35:35.080663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.080636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" event={"ID":"9233d557-a6aa-4d6a-aecd-58cab5f517cd","Type":"ContainerStarted","Data":"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"}
Sep 15 21:35:35.080798 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.080681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" event={"ID":"9233d557-a6aa-4d6a-aecd-58cab5f517cd","Type":"ContainerStarted","Data":"25ed5fca5355c1a2cd314beca77d35205ec3ad87d9b6a3b3889bc20e8294777a"}
Sep 15 21:35:35.081056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.081030    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:35.095274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.095249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:35:35.120046 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.119995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" podStartSLOduration=1.810342587 podStartE2EDuration="2.119974189s" totalImagesPullingTime="309.631602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:35.11995687 +0000 UTC m=+1163.520930118" watchObservedRunningTime="2026-09-15 21:35:35.119974189 +0000 UTC m=+1163.520947436"
Sep 15 21:35:35.157744 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:35.157706    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:35:35.159443 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:35:35.159418    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffc881e_e85e_4250_b7c7_69230edda6b8.slice/crio-49f8ca36a01922c4221fccc9db9555c55de302ecd9bfdab8751974c35e3d7be0 WatchSource:0}: Error finding container 49f8ca36a01922c4221fccc9db9555c55de302ecd9bfdab8751974c35e3d7be0: Status 404 returned error can't find the container with id 49f8ca36a01922c4221fccc9db9555c55de302ecd9bfdab8751974c35e3d7be0
Sep 15 21:35:36.088802 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.088759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" event={"ID":"fffc881e-e85e-4250-b7c7-69230edda6b8","Type":"ContainerStarted","Data":"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"}
Sep 15 21:35:36.089274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.088826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" event={"ID":"fffc881e-e85e-4250-b7c7-69230edda6b8","Type":"ContainerStarted","Data":"49f8ca36a01922c4221fccc9db9555c55de302ecd9bfdab8751974c35e3d7be0"}
Sep 15 21:35:36.089274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.088871    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:36.090553 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.090518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" event={"ID":"f5bd7051-8a23-4306-a4cf-40d95ac44f50","Type":"ContainerStarted","Data":"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"}
Sep 15 21:35:36.107827 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.107770    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" podStartSLOduration=2.107749629 podStartE2EDuration="2.107749629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:36.107426302 +0000 UTC m=+1164.508399547" watchObservedRunningTime="2026-09-15 21:35:36.107749629 +0000 UTC m=+1164.508722876"
Sep 15 21:35:36.131270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:36.131217    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" podStartSLOduration=2.131201149 podStartE2EDuration="2.131201149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:35:36.129861934 +0000 UTC m=+1164.530835199" watchObservedRunningTime="2026-09-15 21:35:36.131201149 +0000 UTC m=+1164.532174412"
Sep 15 21:35:37.099845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:37.099809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:41.096625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:41.096595    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:35:47.101575 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:47.101542    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:35:48.107860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.107828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:35:48.120773 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.120730    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" podUID="7c4131ba-c152-48f1-b848-95efd3408b1c" containerName="main" containerID="cri-o://ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d" gracePeriod=45
Sep 15 21:35:48.370988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.370919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:35:48.449961 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.449925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.449986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450041    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450074    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450146    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwvl\" (UniqueName: \"kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl\") pod \"7c4131ba-c152-48f1-b848-95efd3408b1c\" (UID: \"7c4131ba-c152-48f1-b848-95efd3408b1c\") "
Sep 15 21:35:48.450418 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home" (OuterVolumeSpecName: "home") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:48.450418 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:48.450537 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:48.450660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450641    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:48.450781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450664    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:48.450781 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.450678    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:48.452288 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.452258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm" (OuterVolumeSpecName: "dshm") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:35:48.452420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.452396    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:35:48.452484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.452440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl" (OuterVolumeSpecName: "kube-api-access-cfwvl") pod "7c4131ba-c152-48f1-b848-95efd3408b1c" (UID: "7c4131ba-c152-48f1-b848-95efd3408b1c"). InnerVolumeSpecName "kube-api-access-cfwvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:35:48.552257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.552225    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4131ba-c152-48f1-b848-95efd3408b1c-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:48.552257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.552253    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4131ba-c152-48f1-b848-95efd3408b1c-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:48.552257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:48.552263    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwvl\" (UniqueName: \"kubernetes.io/projected/7c4131ba-c152-48f1-b848-95efd3408b1c-kube-api-access-cfwvl\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:35:49.169233 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.169192    2578 generic.go:356] "Generic (PLEG): container finished" podID="7c4131ba-c152-48f1-b848-95efd3408b1c" containerID="ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d" exitCode=0
Sep 15 21:35:49.169778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.169239    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"
Sep 15 21:35:49.169778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.169274    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" event={"ID":"7c4131ba-c152-48f1-b848-95efd3408b1c","Type":"ContainerDied","Data":"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"}
Sep 15 21:35:49.169778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.169312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds" event={"ID":"7c4131ba-c152-48f1-b848-95efd3408b1c","Type":"ContainerDied","Data":"ca82d862a4de017d3065d4519f4b454b776cfd815adf24c6988e59d41522143b"}
Sep 15 21:35:49.169778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.169330    2578 scope.go:117] "RemoveContainer" containerID="ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"
Sep 15 21:35:49.182012 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.181986    2578 scope.go:117] "RemoveContainer" containerID="ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"
Sep 15 21:35:49.182541 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:35:49.182516    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d\": container with ID starting with ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d not found: ID does not exist" containerID="ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"
Sep 15 21:35:49.182641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.182548    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d"} err="failed to get container status \"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d\": rpc error: code = NotFound desc = could not find container \"ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d\": container with ID starting with ecdbddf798cc65096825a541301a0fa39b1dd18804d896f86460e7d2b239fd3d not found: ID does not exist"
Sep 15 21:35:49.195819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.195793    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:35:49.204018 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:49.203996    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-32270d/explicit-model-32270d-kserve-744c769bb5-db2ds"]
Sep 15 21:35:50.077271 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:35:50.077229    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4131ba-c152-48f1-b848-95efd3408b1c" path="/var/lib/kubelet/pods/7c4131ba-c152-48f1-b848-95efd3408b1c/volumes"
Sep 15 21:36:01.519847 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.519808    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-ea0618"]
Sep 15 21:36:01.520319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.520305    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4131ba-c152-48f1-b848-95efd3408b1c" containerName="main"
Sep 15 21:36:01.520319 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.520319    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4131ba-c152-48f1-b848-95efd3408b1c" containerName="main"
Sep 15 21:36:01.520410 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.520402    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4131ba-c152-48f1-b848-95efd3408b1c" containerName="main"
Sep 15 21:36:01.525228 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.525199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:01.527257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.527236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\""
Sep 15 21:36:01.528153 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.528133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-ea0618"]
Sep 15 21:36:01.589496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.589458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7ln\" (UniqueName: \"kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln\") pod \"test-curl-7751-ea0618\" (UID: \"bf897891-ca08-411a-8ec2-e83f47ec202b\") " pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:01.690943 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.690902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7ln\" (UniqueName: \"kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln\") pod \"test-curl-7751-ea0618\" (UID: \"bf897891-ca08-411a-8ec2-e83f47ec202b\") " pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:01.699539 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.699498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7ln\" (UniqueName: \"kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln\") pod \"test-curl-7751-ea0618\" (UID: \"bf897891-ca08-411a-8ec2-e83f47ec202b\") " pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:01.836914 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:01.836831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:02.109957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.109931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-ea0618"]
Sep 15 21:36:02.112201 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:36:02.112172    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf897891_ca08_411a_8ec2_e83f47ec202b.slice/crio-16a9461a85a73c564d4374ee3c5f3a4083ecc11feaf1bba0372db082c1581185 WatchSource:0}: Error finding container 16a9461a85a73c564d4374ee3c5f3a4083ecc11feaf1bba0372db082c1581185: Status 404 returned error can't find the container with id 16a9461a85a73c564d4374ee3c5f3a4083ecc11feaf1bba0372db082c1581185
Sep 15 21:36:02.245590 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.245546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-ea0618" event={"ID":"bf897891-ca08-411a-8ec2-e83f47ec202b","Type":"ContainerStarted","Data":"8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145"}
Sep 15 21:36:02.245782 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.245598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-ea0618" event={"ID":"bf897891-ca08-411a-8ec2-e83f47ec202b","Type":"ContainerStarted","Data":"16a9461a85a73c564d4374ee3c5f3a4083ecc11feaf1bba0372db082c1581185"}
Sep 15 21:36:02.261440 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.261386    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-ea0618" podStartSLOduration=1.261368429 podStartE2EDuration="1.261368429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:02.259403933 +0000 UTC m=+1190.660377179" watchObservedRunningTime="2026-09-15 21:36:02.261368429 +0000 UTC m=+1190.662341739"
Sep 15 21:36:02.863885 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.863847    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-ea0618"]
Sep 15 21:36:02.868669 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.868635    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-ea0618"]
Sep 15 21:36:02.975752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.975702    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7751-cbacad"]
Sep 15 21:36:02.976235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.976223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" containerName="test-curl-7751-ea0618"
Sep 15 21:36:02.976287 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.976237    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" containerName="test-curl-7751-ea0618"
Sep 15 21:36:02.976327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.976318    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" containerName="test-curl-7751-ea0618"
Sep 15 21:36:02.979580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.979564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:02.986067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:02.986016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-cbacad"]
Sep 15 21:36:03.105232 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:03.105198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkm5z\" (UniqueName: \"kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z\") pod \"test-curl-7751-cbacad\" (UID: \"257a60ef-5755-45f3-8fb2-22761b29f889\") " pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:03.205999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:03.205967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm5z\" (UniqueName: \"kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z\") pod \"test-curl-7751-cbacad\" (UID: \"257a60ef-5755-45f3-8fb2-22761b29f889\") " pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:03.214481 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:03.214449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm5z\" (UniqueName: \"kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z\") pod \"test-curl-7751-cbacad\" (UID: \"257a60ef-5755-45f3-8fb2-22761b29f889\") " pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:03.252170 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:03.252139    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-ea0618" container="test-curl-7751-ea0618"
Sep 15 21:36:03.290402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:03.290370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:03.561569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:03.561538    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7751-cbacad"]
Sep 15 21:36:03.563628 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:36:03.563593    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod257a60ef_5755_45f3_8fb2_22761b29f889.slice/crio-6950e051dfd9681c3f3787cf450d590dbb0398ca5ff95c8364cc9473e82465f7 WatchSource:0}: Error finding container 6950e051dfd9681c3f3787cf450d590dbb0398ca5ff95c8364cc9473e82465f7: Status 404 returned error can't find the container with id 6950e051dfd9681c3f3787cf450d590dbb0398ca5ff95c8364cc9473e82465f7
Sep 15 21:36:04.259364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.259329    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-cbacad" event={"ID":"257a60ef-5755-45f3-8fb2-22761b29f889","Type":"ContainerStarted","Data":"ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8"}
Sep 15 21:36:04.259364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.259370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7751-cbacad" event={"ID":"257a60ef-5755-45f3-8fb2-22761b29f889","Type":"ContainerStarted","Data":"6950e051dfd9681c3f3787cf450d590dbb0398ca5ff95c8364cc9473e82465f7"}
Sep 15 21:36:04.259997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.259371    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-ea0618" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" containerName="test-curl-7751-ea0618" containerID="cri-o://8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145" gracePeriod=2
Sep 15 21:36:04.273684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.273630    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7751-cbacad" podStartSLOduration=2.27361548 podStartE2EDuration="2.27361548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:04.271685304 +0000 UTC m=+1192.672658552" watchObservedRunningTime="2026-09-15 21:36:04.27361548 +0000 UTC m=+1192.674588726"
Sep 15 21:36:04.834327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.834289    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7751-cbacad"]
Sep 15 21:36:04.838338 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:04.838309    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7751-cbacad"]
Sep 15 21:36:05.265279 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:05.265244    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7751-cbacad" container="test-curl-7751-cbacad"
Sep 15 21:36:06.271268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:06.271208    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7751-cbacad" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" containerName="test-curl-7751-cbacad" containerID="cri-o://ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8" gracePeriod=2
Sep 15 21:36:06.455638 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:06.455609    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:06.544318 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:06.544231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz7ln\" (UniqueName: \"kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln\") pod \"bf897891-ca08-411a-8ec2-e83f47ec202b\" (UID: \"bf897891-ca08-411a-8ec2-e83f47ec202b\") "
Sep 15 21:36:06.546284 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:06.546250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln" (OuterVolumeSpecName: "kube-api-access-sz7ln") pod "bf897891-ca08-411a-8ec2-e83f47ec202b" (UID: "bf897891-ca08-411a-8ec2-e83f47ec202b"). InnerVolumeSpecName "kube-api-access-sz7ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:06.645913 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:06.645878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7ln\" (UniqueName: \"kubernetes.io/projected/bf897891-ca08-411a-8ec2-e83f47ec202b-kube-api-access-sz7ln\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:07.277879 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:07.277842    2578 generic.go:356] "Generic (PLEG): container finished" podID="bf897891-ca08-411a-8ec2-e83f47ec202b" containerID="8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145" exitCode=137
Sep 15 21:36:07.278323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:07.277884    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:07.278323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:07.277907    2578 scope.go:117] "RemoveContainer" containerID="8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145"
Sep 15 21:36:07.287476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:07.287448    2578 scope.go:117] "RemoveContainer" containerID="8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145"
Sep 15 21:36:07.287562 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:07.287490    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7751-ea0618\" is forbidden: User \"system:node:ip-10-0-135-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-105.ec2.internal' and this object" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" pod="opendatahub/test-curl-7751-ea0618"
Sep 15 21:36:07.287819 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:07.287800    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145\": container with ID starting with 8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145 not found: ID does not exist" containerID="8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145"
Sep 15 21:36:07.287888 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:07.287828    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145"} err="failed to get container status \"8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145\": rpc error: code = NotFound desc = could not find container \"8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145\": container with ID starting with 8e340a477921b0b36bef0f7802699e01804cd4817e41de818f4a19f0659da145 not found: ID does not exist"
Sep 15 21:36:08.083524 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:08.083487    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf897891-ca08-411a-8ec2-e83f47ec202b" path="/var/lib/kubelet/pods/bf897891-ca08-411a-8ec2-e83f47ec202b/volumes"
Sep 15 21:36:08.457693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:08.457667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:08.563499 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:08.563467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkm5z\" (UniqueName: \"kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z\") pod \"257a60ef-5755-45f3-8fb2-22761b29f889\" (UID: \"257a60ef-5755-45f3-8fb2-22761b29f889\") "
Sep 15 21:36:08.565496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:08.565460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z" (OuterVolumeSpecName: "kube-api-access-rkm5z") pod "257a60ef-5755-45f3-8fb2-22761b29f889" (UID: "257a60ef-5755-45f3-8fb2-22761b29f889"). InnerVolumeSpecName "kube-api-access-rkm5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:08.664688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:08.664607    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkm5z\" (UniqueName: \"kubernetes.io/projected/257a60ef-5755-45f3-8fb2-22761b29f889-kube-api-access-rkm5z\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:09.292191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:09.292157    2578 generic.go:356] "Generic (PLEG): container finished" podID="257a60ef-5755-45f3-8fb2-22761b29f889" containerID="ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8" exitCode=137
Sep 15 21:36:09.292374 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:09.292200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:09.292374 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:09.292221    2578 scope.go:117] "RemoveContainer" containerID="ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8"
Sep 15 21:36:09.302418 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:09.302397    2578 scope.go:117] "RemoveContainer" containerID="ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8"
Sep 15 21:36:09.302630 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:09.302607    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7751-cbacad\" is forbidden: User \"system:node:ip-10-0-135-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-105.ec2.internal' and this object" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" pod="opendatahub/test-curl-7751-cbacad"
Sep 15 21:36:09.302771 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:09.302712    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8\": container with ID starting with ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8 not found: ID does not exist" containerID="ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8"
Sep 15 21:36:09.302771 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:09.302760    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8"} err="failed to get container status \"ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8\": rpc error: code = NotFound desc = could not find container \"ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8\": container with ID starting with ceab881d81f4035c3f28505919713e3e96966da2e166ce1f5cfb49fd0bdabad8 not found: ID does not exist"
Sep 15 21:36:10.076425 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:10.076389    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" path="/var/lib/kubelet/pods/257a60ef-5755-45f3-8fb2-22761b29f889/volumes"
Sep 15 21:36:15.723454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.723421    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:36:15.723955 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.723939    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" containerName="test-curl-7751-cbacad"
Sep 15 21:36:15.723955 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.723956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" containerName="test-curl-7751-cbacad"
Sep 15 21:36:15.724042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.724033    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="257a60ef-5755-45f3-8fb2-22761b29f889" containerName="test-curl-7751-cbacad"
Sep 15 21:36:15.728581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.728563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.731616 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.731588    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"distinct-w0-b99d7b-kserve-self-signed-certs\""
Sep 15 21:36:15.731616 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.731606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"kube-root-ca.crt\""
Sep 15 21:36:15.731832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.731622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"openshift-service-ca.crt\""
Sep 15 21:36:15.731832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.731636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"default-dockercfg-g8wsl\""
Sep 15 21:36:15.737344 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.737467 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb29q\" (UniqueName: \"kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.737533 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.737598 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.737658 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737623    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.737795 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.737746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.740997 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.740977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:36:15.839367 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839563 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb29q\" (UniqueName: \"kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839563 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839563 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839563 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839754 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.839988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.840129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.839962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.840129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.840035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.841862 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.841836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.842036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.842017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:15.847225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:15.847202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb29q\" (UniqueName: \"kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q\") pod \"distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:16.039079 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:16.038995    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:16.325619 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:16.325589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:36:16.328614 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:36:16.328580    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod610adc82_acb1_4b98_877d_0bc6eb5e31db.slice/crio-e82b7a73e3321f2b54e29d431760ec7889bb07b4e19bd0500ae3c7662075e98a WatchSource:0}: Error finding container e82b7a73e3321f2b54e29d431760ec7889bb07b4e19bd0500ae3c7662075e98a: Status 404 returned error can't find the container with id e82b7a73e3321f2b54e29d431760ec7889bb07b4e19bd0500ae3c7662075e98a
Sep 15 21:36:17.338642 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.338602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" event={"ID":"610adc82-acb1-4b98-877d-0bc6eb5e31db","Type":"ContainerStarted","Data":"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"}
Sep 15 21:36:17.339174 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.338653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" event={"ID":"610adc82-acb1-4b98-877d-0bc6eb5e31db","Type":"ContainerStarted","Data":"e82b7a73e3321f2b54e29d431760ec7889bb07b4e19bd0500ae3c7662075e98a"}
Sep 15 21:36:17.387692 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.387639    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" podStartSLOduration=2.387618598 podStartE2EDuration="2.387618598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:17.385934724 +0000 UTC m=+1205.786907970" watchObservedRunningTime="2026-09-15 21:36:17.387618598 +0000 UTC m=+1205.788591846"
Sep 15 21:36:17.541887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.541849    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:36:17.550697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.547345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.551059 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.551033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-56820a3c\""
Sep 15 21:36:17.551183 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.551076    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-56820a3c-dockercfg-mr2rs\""
Sep 15 21:36:17.554930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.554902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzw8\" (UniqueName: \"kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.555058 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.555009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.578090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.578053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:36:17.656527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.656426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzw8\" (UniqueName: \"kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.656687 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.656576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.657523 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.657493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.664621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.664591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzw8\" (UniqueName: \"kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8\") pod \"payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:17.861742 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:17.861677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:18.148148 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:18.148122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:36:18.150536 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:36:18.150511    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6988b56d_91a0_430d_94b7_9b1fbe763ee1.slice/crio-bfcc105f37f6496dd5a024313531601231e603e481d3c58b450cea5e135fcde7 WatchSource:0}: Error finding container bfcc105f37f6496dd5a024313531601231e603e481d3c58b450cea5e135fcde7: Status 404 returned error can't find the container with id bfcc105f37f6496dd5a024313531601231e603e481d3c58b450cea5e135fcde7
Sep 15 21:36:18.347431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:18.347390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" event={"ID":"6988b56d-91a0-430d-94b7-9b1fbe763ee1","Type":"ContainerStarted","Data":"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"}
Sep 15 21:36:18.347902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:18.347591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:18.347902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:18.347624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" event={"ID":"6988b56d-91a0-430d-94b7-9b1fbe763ee1","Type":"ContainerStarted","Data":"bfcc105f37f6496dd5a024313531601231e603e481d3c58b450cea5e135fcde7"}
Sep 15 21:36:18.368300 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:18.368244    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" podStartSLOduration=1.368228094 podStartE2EDuration="1.368228094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:18.365852926 +0000 UTC m=+1206.766826173" watchObservedRunningTime="2026-09-15 21:36:18.368228094 +0000 UTC m=+1206.769201339"
Sep 15 21:36:22.101112 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:22.101071    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:36:26.039829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:26.039792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:26.039829 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:26.039836    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:26.053141 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:26.053115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:26.403660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:26.403575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:36:27.891547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.891505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:36:27.895838 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.895812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.897949 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.897924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:36:27.898094 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.897961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4a69fe-openshift-default-dockercfg-svlnr\""
Sep 15 21:36:27.909351 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.909318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:36:27.968576 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968576 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fs8q\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.968833 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.969003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:27.969003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:27.968916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070148 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070148 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fs8q\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070419 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.070841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070753    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.071018 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.070995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.071096 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.071022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.071142 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.071102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.072857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.072834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.073005 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.072989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.078036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.078013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fs8q\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.078376 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.078357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token\") pod \"e2e-noten-4a69fe-openshift-default-fc499649d-d27ql\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") " pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.208070 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.208035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:28.497876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.497844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:36:28.499738 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:36:28.499693    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49eccd57_3c23_4488_bf85_db8612c82fa7.slice/crio-d64deb9e24a66a099d00ad95f776b96c90349a949e60536eb2cc5f95668ef2ff WatchSource:0}: Error finding container d64deb9e24a66a099d00ad95f776b96c90349a949e60536eb2cc5f95668ef2ff: Status 404 returned error can't find the container with id d64deb9e24a66a099d00ad95f776b96c90349a949e60536eb2cc5f95668ef2ff
Sep 15 21:36:28.502285 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.502245    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:36:28.502400 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.502318    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:36:28.502400 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:28.502356    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:36:29.355621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:29.355590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:36:29.411761 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:29.411709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" event={"ID":"49eccd57-3c23-4488-bf85-db8612c82fa7","Type":"ContainerStarted","Data":"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"}
Sep 15 21:36:29.411906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:29.411782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" event={"ID":"49eccd57-3c23-4488-bf85-db8612c82fa7","Type":"ContainerStarted","Data":"d64deb9e24a66a099d00ad95f776b96c90349a949e60536eb2cc5f95668ef2ff"}
Sep 15 21:36:29.432965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:29.432897    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" podStartSLOduration=2.432874853 podStartE2EDuration="2.432874853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:29.429091982 +0000 UTC m=+1217.830065228" watchObservedRunningTime="2026-09-15 21:36:29.432874853 +0000 UTC m=+1217.833848099"
Sep 15 21:36:30.208977 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:30.208940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:30.214881 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:30.214853    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:30.417968 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:30.417938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:30.418929 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:30.418910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:36:35.971587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:35.971549    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:36:35.972283 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:35.971809    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" podUID="bd271b92-b2ab-4e39-a393-8e71066c82f1" containerName="maas-api" containerID="cri-o://9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f" gracePeriod=30
Sep 15 21:36:36.035594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.035549    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:36:36.035814 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.035789    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" podUID="228752dd-c92f-4307-9977-4fd9a38f880a" containerName="payload-pre-processing" containerID="cri-o://53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929" gracePeriod=30
Sep 15 21:36:36.339932 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.339901    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:36:36.343354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.343328    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:36:36.451354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.451323    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd271b92-b2ab-4e39-a393-8e71066c82f1" containerID="9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f" exitCode=0
Sep 15 21:36:36.451562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.451375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"
Sep 15 21:36:36.451562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.451411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" event={"ID":"bd271b92-b2ab-4e39-a393-8e71066c82f1","Type":"ContainerDied","Data":"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"}
Sep 15 21:36:36.451562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.451461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc" event={"ID":"bd271b92-b2ab-4e39-a393-8e71066c82f1","Type":"ContainerDied","Data":"761c546847bb9065dc706408df91a19edf170d10fde79f23d50d1f84fd47de46"}
Sep 15 21:36:36.451562 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.451475    2578 scope.go:117] "RemoveContainer" containerID="9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"
Sep 15 21:36:36.452945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.452922    2578 generic.go:356] "Generic (PLEG): container finished" podID="228752dd-c92f-4307-9977-4fd9a38f880a" containerID="53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929" exitCode=0
Sep 15 21:36:36.453053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.452965    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"
Sep 15 21:36:36.453053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.452999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" event={"ID":"228752dd-c92f-4307-9977-4fd9a38f880a","Type":"ContainerDied","Data":"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"}
Sep 15 21:36:36.453053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.453040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp" event={"ID":"228752dd-c92f-4307-9977-4fd9a38f880a","Type":"ContainerDied","Data":"a629fc402fbe94b46256e89f4d5e7db4d72c51adf976ab5fa628dcb30b85cd3c"}
Sep 15 21:36:36.460681 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.460664    2578 scope.go:117] "RemoveContainer" containerID="9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"
Sep 15 21:36:36.460977 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:36.460957    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f\": container with ID starting with 9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f not found: ID does not exist" containerID="9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"
Sep 15 21:36:36.461085 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.460984    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f"} err="failed to get container status \"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f\": rpc error: code = NotFound desc = could not find container \"9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f\": container with ID starting with 9e50c00c20c2de4d8fa010ac4e1bbeb94f94d0355a6c818cb6fc4383c69a535f not found: ID does not exist"
Sep 15 21:36:36.461085 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.461000    2578 scope.go:117] "RemoveContainer" containerID="53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"
Sep 15 21:36:36.466345 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzfqm\" (UniqueName: \"kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm\") pod \"bd271b92-b2ab-4e39-a393-8e71066c82f1\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") "
Sep 15 21:36:36.466430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs\") pod \"bd271b92-b2ab-4e39-a393-8e71066c82f1\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") "
Sep 15 21:36:36.466430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c27c9\" (UniqueName: \"kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9\") pod \"228752dd-c92f-4307-9977-4fd9a38f880a\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") "
Sep 15 21:36:36.466535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls\") pod \"bd271b92-b2ab-4e39-a393-8e71066c82f1\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") "
Sep 15 21:36:36.466535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca\") pod \"bd271b92-b2ab-4e39-a393-8e71066c82f1\" (UID: \"bd271b92-b2ab-4e39-a393-8e71066c82f1\") "
Sep 15 21:36:36.466654 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.466561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume\") pod \"228752dd-c92f-4307-9977-4fd9a38f880a\" (UID: \"228752dd-c92f-4307-9977-4fd9a38f880a\") "
Sep 15 21:36:36.467097 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.467041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "228752dd-c92f-4307-9977-4fd9a38f880a" (UID: "228752dd-c92f-4307-9977-4fd9a38f880a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:36.467206 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.467170    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228752dd-c92f-4307-9977-4fd9a38f880a-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.467590 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.467561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd271b92-b2ab-4e39-a393-8e71066c82f1" (UID: "bd271b92-b2ab-4e39-a393-8e71066c82f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:36.468637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.468600    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm" (OuterVolumeSpecName: "kube-api-access-wzfqm") pod "bd271b92-b2ab-4e39-a393-8e71066c82f1" (UID: "bd271b92-b2ab-4e39-a393-8e71066c82f1"). InnerVolumeSpecName "kube-api-access-wzfqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:36.468752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.468694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd271b92-b2ab-4e39-a393-8e71066c82f1" (UID: "bd271b92-b2ab-4e39-a393-8e71066c82f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:36.468887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.468870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9" (OuterVolumeSpecName: "kube-api-access-c27c9") pod "228752dd-c92f-4307-9977-4fd9a38f880a" (UID: "228752dd-c92f-4307-9977-4fd9a38f880a"). InnerVolumeSpecName "kube-api-access-c27c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:36.468958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.468911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd271b92-b2ab-4e39-a393-8e71066c82f1" (UID: "bd271b92-b2ab-4e39-a393-8e71066c82f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:36.471126 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.471107    2578 scope.go:117] "RemoveContainer" containerID="53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"
Sep 15 21:36:36.471428 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:36.471411    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929\": container with ID starting with 53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929 not found: ID does not exist" containerID="53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"
Sep 15 21:36:36.471489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.471436    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929"} err="failed to get container status \"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929\": rpc error: code = NotFound desc = could not find container \"53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929\": container with ID starting with 53c0537173c7273f4eb7328738499662b4dcdcca74eb992c5d6ce886c7bb9929 not found: ID does not exist"
Sep 15 21:36:36.568546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.568456    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfqm\" (UniqueName: \"kubernetes.io/projected/bd271b92-b2ab-4e39-a393-8e71066c82f1-kube-api-access-wzfqm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.568546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.568499    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.568546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.568514    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27c9\" (UniqueName: \"kubernetes.io/projected/228752dd-c92f-4307-9977-4fd9a38f880a-kube-api-access-c27c9\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.568546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.568527    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd271b92-b2ab-4e39-a393-8e71066c82f1-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.568546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.568540    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd271b92-b2ab-4e39-a393-8e71066c82f1-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:36.794171 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.794139    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:36:36.804845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.804807    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4bc420-6f6ccfb88b-tpjwp"]
Sep 15 21:36:36.826396 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.826309    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:36:36.831837 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:36.831812    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4bc420-85b886f544-8fqlc"]
Sep 15 21:36:37.122493 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.122400    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" podUID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" containerName="main" containerID="cri-o://90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468" gracePeriod=45
Sep 15 21:36:37.403741 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.403497    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:36:37.483120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.483120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.483464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.483464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.483464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.483464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.483323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache\") pod \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\" (UID: \"9d01ae12-81d0-4d1b-935b-afc5c4018c2f\") "
Sep 15 21:36:37.495869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.484680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home" (OuterVolumeSpecName: "home") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:37.495869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.486237    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:37.495869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.493790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:37.498966 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.498512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm" (OuterVolumeSpecName: "dshm") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:36:37.499132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.498961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:36:37.502828 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.499302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf" (OuterVolumeSpecName: "kube-api-access-b9nqf") pod "9d01ae12-81d0-4d1b-935b-afc5c4018c2f" (UID: "9d01ae12-81d0-4d1b-935b-afc5c4018c2f"). InnerVolumeSpecName "kube-api-access-b9nqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:37.518517 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.518476    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" containerID="90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468" exitCode=0
Sep 15 21:36:37.518688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.518538    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"
Sep 15 21:36:37.518688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.518550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" event={"ID":"9d01ae12-81d0-4d1b-935b-afc5c4018c2f","Type":"ContainerDied","Data":"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"}
Sep 15 21:36:37.518688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.518590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49" event={"ID":"9d01ae12-81d0-4d1b-935b-afc5c4018c2f","Type":"ContainerDied","Data":"3d189e688941518e53e8eeb2b385c5a28629578957a716ee553dc2c6a9be0d40"}
Sep 15 21:36:37.518688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.518608    2578 scope.go:117] "RemoveContainer" containerID="90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"
Sep 15 21:36:37.529779 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.529712    2578 scope.go:117] "RemoveContainer" containerID="90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"
Sep 15 21:36:37.530116 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:37.530093    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468\": container with ID starting with 90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468 not found: ID does not exist" containerID="90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"
Sep 15 21:36:37.530214 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.530127    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468"} err="failed to get container status \"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468\": rpc error: code = NotFound desc = could not find container \"90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468\": container with ID starting with 90045b980de0b31cfca42d79b9fff99d4af64e79eda7e596b478732a4ce3e468 not found: ID does not exist"
Sep 15 21:36:37.540752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.540704    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:36:37.543006 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.542984    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4bc420/test-model-4bc420-kserve-8456d75955-plw49"]
Sep 15 21:36:37.585177 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585138    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nqf\" (UniqueName: \"kubernetes.io/projected/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-kube-api-access-b9nqf\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.585177 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585177    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.585397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585192    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.585397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585231    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.585397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585245    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:37.585397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:37.585255    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d01ae12-81d0-4d1b-935b-afc5c4018c2f-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:38.077976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:38.077935    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228752dd-c92f-4307-9977-4fd9a38f880a" path="/var/lib/kubelet/pods/228752dd-c92f-4307-9977-4fd9a38f880a/volumes"
Sep 15 21:36:38.078499 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:38.078474    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" path="/var/lib/kubelet/pods/9d01ae12-81d0-4d1b-935b-afc5c4018c2f/volumes"
Sep 15 21:36:38.078979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:38.078957    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd271b92-b2ab-4e39-a393-8e71066c82f1" path="/var/lib/kubelet/pods/bd271b92-b2ab-4e39-a393-8e71066c82f1/volumes"
Sep 15 21:36:54.039625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.038462    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:36:54.039625 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.038750    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" podUID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" containerName="payload-pre-processing" containerID="cri-o://c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000" gracePeriod=30
Sep 15 21:36:54.299033 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.298967    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:36:54.359398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.359359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqdkp\" (UniqueName: \"kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp\") pod \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") "
Sep 15 21:36:54.359592 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.359499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume\") pod \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\" (UID: \"52ac5b02-9cc5-4ea6-bff7-f17281469e7f\") "
Sep 15 21:36:54.359930 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.359895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52ac5b02-9cc5-4ea6-bff7-f17281469e7f" (UID: "52ac5b02-9cc5-4ea6-bff7-f17281469e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:36:54.361596 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.361565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp" (OuterVolumeSpecName: "kube-api-access-mqdkp") pod "52ac5b02-9cc5-4ea6-bff7-f17281469e7f" (UID: "52ac5b02-9cc5-4ea6-bff7-f17281469e7f"). InnerVolumeSpecName "kube-api-access-mqdkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:36:54.461291 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.461253    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:54.461291 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.461284    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdkp\" (UniqueName: \"kubernetes.io/projected/52ac5b02-9cc5-4ea6-bff7-f17281469e7f-kube-api-access-mqdkp\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:36:54.627193 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.627103    2578 generic.go:356] "Generic (PLEG): container finished" podID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" containerID="c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000" exitCode=0
Sep 15 21:36:54.627193 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.627138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" event={"ID":"52ac5b02-9cc5-4ea6-bff7-f17281469e7f","Type":"ContainerDied","Data":"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"}
Sep 15 21:36:54.627193 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.627155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"
Sep 15 21:36:54.627521 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.627201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg" event={"ID":"52ac5b02-9cc5-4ea6-bff7-f17281469e7f","Type":"ContainerDied","Data":"bb7199e731c3ed5458612f7791b86210d36f5152e1661dddb3b9b93aa349d842"}
Sep 15 21:36:54.627521 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.627228    2578 scope.go:117] "RemoveContainer" containerID="c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"
Sep 15 21:36:54.637132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.637112    2578 scope.go:117] "RemoveContainer" containerID="c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"
Sep 15 21:36:54.637436 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:36:54.637414    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000\": container with ID starting with c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000 not found: ID does not exist" containerID="c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"
Sep 15 21:36:54.637501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.637444    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000"} err="failed to get container status \"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000\": rpc error: code = NotFound desc = could not find container \"c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000\": container with ID starting with c7a538a16b86221d3ec502205a10f08cbac4eea300295a6ed9a22298d19bd000 not found: ID does not exist"
Sep 15 21:36:54.650327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.650301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:36:54.654652 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:54.654623    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bb7867-74958948f8-jmfvg"]
Sep 15 21:36:56.076945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:36:56.076912    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" path="/var/lib/kubelet/pods/52ac5b02-9cc5-4ea6-bff7-f17281469e7f/volumes"
Sep 15 21:37:23.276485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276447    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:37:23.276948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276934    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228752dd-c92f-4307-9977-4fd9a38f880a" containerName="payload-pre-processing"
Sep 15 21:37:23.276995 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276950    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="228752dd-c92f-4307-9977-4fd9a38f880a" containerName="payload-pre-processing"
Sep 15 21:37:23.276995 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd271b92-b2ab-4e39-a393-8e71066c82f1" containerName="maas-api"
Sep 15 21:37:23.276995 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276981    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd271b92-b2ab-4e39-a393-8e71066c82f1" containerName="maas-api"
Sep 15 21:37:23.276995 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276993    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" containerName="payload-pre-processing"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.276999    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" containerName="payload-pre-processing"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" containerName="main"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277010    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" containerName="main"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277083    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="228752dd-c92f-4307-9977-4fd9a38f880a" containerName="payload-pre-processing"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277093    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d01ae12-81d0-4d1b-935b-afc5c4018c2f" containerName="main"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277100    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd271b92-b2ab-4e39-a393-8e71066c82f1" containerName="maas-api"
Sep 15 21:37:23.277125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.277107    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ac5b02-9cc5-4ea6-bff7-f17281469e7f" containerName="payload-pre-processing"
Sep 15 21:37:23.282785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.282760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.284672 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.284650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-59e243-dockercfg-wcfqp\""
Sep 15 21:37:23.284834 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.284658    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-59e243\""
Sep 15 21:37:23.292066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.292039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:37:23.384906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.384866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzkp\" (UniqueName: \"kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.385081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.384926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.485994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.485939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzkp\" (UniqueName: \"kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.486222 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.486005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.486792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.486765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.494463 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.494435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzkp\" (UniqueName: \"kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp\") pod \"payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.593580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.593499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:23.864549 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:23.864523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:37:23.866595 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:37:23.866567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51f7c3f_379e_49a8_8dd5_402573ea6c4b.slice/crio-c4878dc431d47ba38c017ecae5b16d97b20291dff25a7771b68f769827f1e609 WatchSource:0}: Error finding container c4878dc431d47ba38c017ecae5b16d97b20291dff25a7771b68f769827f1e609: Status 404 returned error can't find the container with id c4878dc431d47ba38c017ecae5b16d97b20291dff25a7771b68f769827f1e609
Sep 15 21:37:24.813133 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:24.813082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" event={"ID":"a51f7c3f-379e-49a8-8dd5-402573ea6c4b","Type":"ContainerStarted","Data":"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"}
Sep 15 21:37:24.813569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:24.813151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" event={"ID":"a51f7c3f-379e-49a8-8dd5-402573ea6c4b","Type":"ContainerStarted","Data":"c4878dc431d47ba38c017ecae5b16d97b20291dff25a7771b68f769827f1e609"}
Sep 15 21:37:24.813569 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:24.813304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:24.831925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:24.831846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" podStartSLOduration=1.831825853 podStartE2EDuration="1.831825853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:24.828333539 +0000 UTC m=+1273.229306786" watchObservedRunningTime="2026-09-15 21:37:24.831825853 +0000 UTC m=+1273.232799101"
Sep 15 21:37:35.821819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:35.821783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:37:59.069814 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.069777    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:37:59.073804 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.073778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.076933 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.076909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-38a3a3-dockercfg-nd7d4\""
Sep 15 21:37:59.077069 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.076986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-38a3a3\""
Sep 15 21:37:59.091246 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.091212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:37:59.126545 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.126505    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:37:59.135999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.135959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.144639 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.144605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:37:59.157244 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.157199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.157418 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.157344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz98r\" (UniqueName: \"kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.157487 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.157419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.157568 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.157494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42pp\" (UniqueName: \"kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.258444 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.258404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.258623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.258492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz98r\" (UniqueName: \"kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.258623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.258536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.258623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.258593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42pp\" (UniqueName: \"kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.259187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.259164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.259346 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.259323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.268358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.268320    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz98r\" (UniqueName: \"kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r\") pod \"payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.268547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.268529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42pp\" (UniqueName: \"kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp\") pod \"payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.385195 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.385109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:37:59.450483 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.450162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:37:59.658451 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.658425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:37:59.659962 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:37:59.659930    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9670021_907d_4628_a067_992e8367063a.slice/crio-90ad368d78680c505c402fb97c027a72f018a2268c65418f1ab08ff9a4395581 WatchSource:0}: Error finding container 90ad368d78680c505c402fb97c027a72f018a2268c65418f1ab08ff9a4395581: Status 404 returned error can't find the container with id 90ad368d78680c505c402fb97c027a72f018a2268c65418f1ab08ff9a4395581
Sep 15 21:37:59.733420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:37:59.733395    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:37:59.734953 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:37:59.734928    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fc4ca5_bf8a_4d5f_a1ae_945962a26d88.slice/crio-8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba WatchSource:0}: Error finding container 8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba: Status 404 returned error can't find the container with id 8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba
Sep 15 21:38:00.014143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.014100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" event={"ID":"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88","Type":"ContainerStarted","Data":"4516efe1dc092ad21d3cb5b34ae0d9c1b68535a41676aa7ff20bd0bc35e8ba50"}
Sep 15 21:38:00.014358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.014173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" event={"ID":"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88","Type":"ContainerStarted","Data":"8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba"}
Sep 15 21:38:00.014358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.014295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:38:00.016214 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.016187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" event={"ID":"b9670021-907d-4628-a067-992e8367063a","Type":"ContainerStarted","Data":"cac6da06fb79e59ae37ff4f3812ae45ed0e5daa5af1d4eccd2c3b214cb2aaa5e"}
Sep 15 21:38:00.016359 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.016232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" event={"ID":"b9670021-907d-4628-a067-992e8367063a","Type":"ContainerStarted","Data":"90ad368d78680c505c402fb97c027a72f018a2268c65418f1ab08ff9a4395581"}
Sep 15 21:38:00.016359 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.016259    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:38:00.029921 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.029878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" podStartSLOduration=1.029863631 podStartE2EDuration="1.029863631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:00.028683289 +0000 UTC m=+1308.429656536" watchObservedRunningTime="2026-09-15 21:38:00.029863631 +0000 UTC m=+1308.430836877"
Sep 15 21:38:00.043664 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:00.043597    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" podStartSLOduration=1.043579079 podStartE2EDuration="1.043579079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:00.043406824 +0000 UTC m=+1308.444380073" watchObservedRunningTime="2026-09-15 21:38:00.043579079 +0000 UTC m=+1308.444552325"
Sep 15 21:38:03.339632 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:03.339598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:38:03.340057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:03.339818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" podUID="49eccd57-3c23-4488-bf85-db8612c82fa7" containerName="istio-proxy" containerID="cri-o://89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f" gracePeriod=30
Sep 15 21:38:08.599038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.598971    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:38:08.657819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.657774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.657819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.657825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.657872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.657905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fs8q\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.657997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658162    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy\") pod \"49eccd57-3c23-4488-bf85-db8612c82fa7\" (UID: \"49eccd57-3c23-4488-bf85-db8612c82fa7\") "
Sep 15 21:38:08.658302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658068    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:08.658527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:08.658527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:08.658527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:38:08.658527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658509    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data" (OuterVolumeSpecName: "istio-data") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:08.658751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658665    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49eccd57-3c23-4488-bf85-db8612c82fa7-istiod-ca-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.658751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658686    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-credential-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.658751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658700    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-data\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.658751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658734    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.658969 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.658753    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-workload-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.660146 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.660121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q" (OuterVolumeSpecName: "kube-api-access-5fs8q") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "kube-api-access-5fs8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:08.660265 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.660253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token" (OuterVolumeSpecName: "istio-token") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:38:08.660352 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.660336    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:38:08.660680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.660652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49eccd57-3c23-4488-bf85-db8612c82fa7" (UID: "49eccd57-3c23-4488-bf85-db8612c82fa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:38:08.759743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.759680    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-token\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.759743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.759728    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-envoy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.759743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.759738    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49eccd57-3c23-4488-bf85-db8612c82fa7-istio-podinfo\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:08.759743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:08.759748    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fs8q\" (UniqueName: \"kubernetes.io/projected/49eccd57-3c23-4488-bf85-db8612c82fa7-kube-api-access-5fs8q\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:38:09.066204 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.066169    2578 generic.go:356] "Generic (PLEG): container finished" podID="49eccd57-3c23-4488-bf85-db8612c82fa7" containerID="89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f" exitCode=0
Sep 15 21:38:09.066379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.066211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"
Sep 15 21:38:09.066379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.066232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" event={"ID":"49eccd57-3c23-4488-bf85-db8612c82fa7","Type":"ContainerDied","Data":"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"}
Sep 15 21:38:09.066379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.066283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql" event={"ID":"49eccd57-3c23-4488-bf85-db8612c82fa7","Type":"ContainerDied","Data":"d64deb9e24a66a099d00ad95f776b96c90349a949e60536eb2cc5f95668ef2ff"}
Sep 15 21:38:09.066379 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.066308    2578 scope.go:117] "RemoveContainer" containerID="89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"
Sep 15 21:38:09.076183 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.076160    2578 scope.go:117] "RemoveContainer" containerID="89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"
Sep 15 21:38:09.076458 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:38:09.076438    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f\": container with ID starting with 89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f not found: ID does not exist" containerID="89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"
Sep 15 21:38:09.076500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.076470    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f"} err="failed to get container status \"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f\": rpc error: code = NotFound desc = could not find container \"89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f\": container with ID starting with 89096d3f31159f914f6186d185be687dd4c89e9963613ec40a5bbebf52391f3f not found: ID does not exist"
Sep 15 21:38:09.090093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.090061    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:38:09.094753 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:09.094726    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4a69fe-openshift-default-fc499649d-d27ql"]
Sep 15 21:38:10.076381 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:10.076345    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49eccd57-3c23-4488-bf85-db8612c82fa7" path="/var/lib/kubelet/pods/49eccd57-3c23-4488-bf85-db8612c82fa7/volumes"
Sep 15 21:38:11.023710 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:11.023680    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:38:11.023925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:11.023773    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:38:37.122003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.121922    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:38:37.122492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.122461    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49eccd57-3c23-4488-bf85-db8612c82fa7" containerName="istio-proxy"
Sep 15 21:38:37.122492 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.122478    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49eccd57-3c23-4488-bf85-db8612c82fa7" containerName="istio-proxy"
Sep 15 21:38:37.122594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.122574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49eccd57-3c23-4488-bf85-db8612c82fa7" containerName="istio-proxy"
Sep 15 21:38:37.127132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.127102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.129750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.129707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b99d7b\"/\"distinct-2-w0-b99d7b-kserve-self-signed-certs\""
Sep 15 21:38:37.140428 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.140401    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:38:37.263179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.263179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.263407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.263407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.263488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrw4n\" (UniqueName: \"kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.263488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.263442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364318    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrw4n\" (UniqueName: \"kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364526    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.364976 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.365032 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.364974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.365032 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.365008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.366688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.366661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.367090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.367067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.371696 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.371677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrw4n\" (UniqueName: \"kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n\") pod \"distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") " pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.438299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.438267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:37.714132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:37.713982    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:38:37.716538 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:38:37.716507    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3514f173_0573_415b_8682_f6c971179d1e.slice/crio-da72328d57110de72be131f7268bf12097849afcdbe93063b6b869fc1a53c367 WatchSource:0}: Error finding container da72328d57110de72be131f7268bf12097849afcdbe93063b6b869fc1a53c367: Status 404 returned error can't find the container with id da72328d57110de72be131f7268bf12097849afcdbe93063b6b869fc1a53c367
Sep 15 21:38:38.232902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:38.232864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" event={"ID":"3514f173-0573-415b-8682-f6c971179d1e","Type":"ContainerStarted","Data":"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"}
Sep 15 21:38:38.233290 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:38.232909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" event={"ID":"3514f173-0573-415b-8682-f6c971179d1e","Type":"ContainerStarted","Data":"da72328d57110de72be131f7268bf12097849afcdbe93063b6b869fc1a53c367"}
Sep 15 21:38:38.263612 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:38.263551    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" podStartSLOduration=1.26353557 podStartE2EDuration="1.26353557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:38:38.260888304 +0000 UTC m=+1346.661861549" watchObservedRunningTime="2026-09-15 21:38:38.26353557 +0000 UTC m=+1346.664508812"
Sep 15 21:38:47.439041 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:47.439000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:47.439522 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:47.439063    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:47.453249 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:47.453222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:38:48.305330 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:38:48.305292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:39:13.106240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.106206    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:39:13.109966 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.109937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.112190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.112161    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"kube-root-ca.crt\""
Sep 15 21:39:13.112327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.112219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"openshift-service-ca.crt\""
Sep 15 21:39:13.112327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.112161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"auth-test-model-59e243-kserve-self-signed-certs\""
Sep 15 21:39:13.112327 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.112163    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"default-dockercfg-zpj9x\""
Sep 15 21:39:13.117483 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.117458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:39:13.264368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.264546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.264546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmkc\" (UniqueName: \"kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.264546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.264546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.264782 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.264607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.365784 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.365784 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmkc\" (UniqueName: \"kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.365784 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.365784 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.366104 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.366104 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.365983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.366104 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.366089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.366270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.366188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.366329 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.366274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.368019 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.368000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.368303 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.368285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.373988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.373967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmkc\" (UniqueName: \"kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc\") pod \"auth-test-model-59e243-kserve-76b64b4db7-p42ks\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") " pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.421822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.421780    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:13.695484 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:13.695456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:39:13.697076 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:39:13.697053    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47e4ba9c_9173_4444_a442_ad170c76fdbe.slice/crio-3a38c8996847f3b70129a3ff194daf7e6bab5369da177de488fbf3a073c92cc5 WatchSource:0}: Error finding container 3a38c8996847f3b70129a3ff194daf7e6bab5369da177de488fbf3a073c92cc5: Status 404 returned error can't find the container with id 3a38c8996847f3b70129a3ff194daf7e6bab5369da177de488fbf3a073c92cc5
Sep 15 21:39:14.444227 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:14.444187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" event={"ID":"47e4ba9c-9173-4444-a442-ad170c76fdbe","Type":"ContainerStarted","Data":"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"}
Sep 15 21:39:14.444227 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:14.444229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" event={"ID":"47e4ba9c-9173-4444-a442-ad170c76fdbe","Type":"ContainerStarted","Data":"3a38c8996847f3b70129a3ff194daf7e6bab5369da177de488fbf3a073c92cc5"}
Sep 15 21:39:14.460688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:14.460632    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" podStartSLOduration=1.46061294 podStartE2EDuration="1.46061294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:14.460527294 +0000 UTC m=+1382.861500539" watchObservedRunningTime="2026-09-15 21:39:14.46061294 +0000 UTC m=+1382.861586187"
Sep 15 21:39:23.422139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:23.422091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:23.422139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:23.422147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:23.435224 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:23.435197    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:23.506690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:23.506660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:39:34.167374 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.167325    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:39:34.167882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.167587    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" podUID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" containerName="payload-pre-processing" containerID="cri-o://63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896" gracePeriod=30
Sep 15 21:39:34.432253 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.432186    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:39:34.491795 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.491761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume\") pod \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") "
Sep 15 21:39:34.491994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.491839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgzw8\" (UniqueName: \"kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8\") pod \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\" (UID: \"6988b56d-91a0-430d-94b7-9b1fbe763ee1\") "
Sep 15 21:39:34.492220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.492193    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6988b56d-91a0-430d-94b7-9b1fbe763ee1" (UID: "6988b56d-91a0-430d-94b7-9b1fbe763ee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:39:34.492349 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.492333    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6988b56d-91a0-430d-94b7-9b1fbe763ee1-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:39:34.494057 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.494031    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8" (OuterVolumeSpecName: "kube-api-access-sgzw8") pod "6988b56d-91a0-430d-94b7-9b1fbe763ee1" (UID: "6988b56d-91a0-430d-94b7-9b1fbe763ee1"). InnerVolumeSpecName "kube-api-access-sgzw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:39:34.562627 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.562590    2578 generic.go:356] "Generic (PLEG): container finished" podID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" containerID="63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896" exitCode=0
Sep 15 21:39:34.562843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.562632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"
Sep 15 21:39:34.562843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.562676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" event={"ID":"6988b56d-91a0-430d-94b7-9b1fbe763ee1","Type":"ContainerDied","Data":"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"}
Sep 15 21:39:34.562843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.562753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp" event={"ID":"6988b56d-91a0-430d-94b7-9b1fbe763ee1","Type":"ContainerDied","Data":"bfcc105f37f6496dd5a024313531601231e603e481d3c58b450cea5e135fcde7"}
Sep 15 21:39:34.562843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.562787    2578 scope.go:117] "RemoveContainer" containerID="63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"
Sep 15 21:39:34.581363 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.581318    2578 scope.go:117] "RemoveContainer" containerID="63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"
Sep 15 21:39:34.581980 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:39:34.581948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896\": container with ID starting with 63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896 not found: ID does not exist" containerID="63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"
Sep 15 21:39:34.582095 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.581989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896"} err="failed to get container status \"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896\": rpc error: code = NotFound desc = could not find container \"63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896\": container with ID starting with 63a25c25bd8735ff5b3059078a365166a8d55510c48029e50ec553a7bb8f7896 not found: ID does not exist"
Sep 15 21:39:34.586863 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.586595    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:39:34.590387 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.590362    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-56820a3c-5dcc7bf678-zf5vp"]
Sep 15 21:39:34.593207 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:34.593181    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgzw8\" (UniqueName: \"kubernetes.io/projected/6988b56d-91a0-430d-94b7-9b1fbe763ee1-kube-api-access-sgzw8\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:39:36.076391 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:36.076357    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" path="/var/lib/kubelet/pods/6988b56d-91a0-430d-94b7-9b1fbe763ee1/volumes"
Sep 15 21:39:51.135923 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:39:51.135830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:40:06.156547 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.156457    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" podUID="b1859a80-4fe5-4230-a057-bf74f4c56af6" containerName="main" containerID="cri-o://be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0" gracePeriod=45
Sep 15 21:40:06.409396 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.409331    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:40:06.532476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532476 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532626    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfv5s\" (UniqueName: \"kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532666    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm\") pod \"b1859a80-4fe5-4230-a057-bf74f4c56af6\" (UID: \"b1859a80-4fe5-4230-a057-bf74f4c56af6\") "
Sep 15 21:40:06.532990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:06.532990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:06.532990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.532860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home" (OuterVolumeSpecName: "home") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:06.533176 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.533156    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.533235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.533180    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.533235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.533193    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.534999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.534974    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm" (OuterVolumeSpecName: "dshm") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:06.535116 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.534996    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s" (OuterVolumeSpecName: "kube-api-access-rfv5s") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "kube-api-access-rfv5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:06.535116 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.534997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1859a80-4fe5-4230-a057-bf74f4c56af6" (UID: "b1859a80-4fe5-4230-a057-bf74f4c56af6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:40:06.633792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.633756    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfv5s\" (UniqueName: \"kubernetes.io/projected/b1859a80-4fe5-4230-a057-bf74f4c56af6-kube-api-access-rfv5s\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.633792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.633786    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1859a80-4fe5-4230-a057-bf74f4c56af6-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.633792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.633797    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1859a80-4fe5-4230-a057-bf74f4c56af6-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.744171 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.744141    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1859a80-4fe5-4230-a057-bf74f4c56af6" containerID="be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0" exitCode=0
Sep 15 21:40:06.744380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.744195    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"
Sep 15 21:40:06.744380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.744211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" event={"ID":"b1859a80-4fe5-4230-a057-bf74f4c56af6","Type":"ContainerDied","Data":"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"}
Sep 15 21:40:06.744380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.744247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs" event={"ID":"b1859a80-4fe5-4230-a057-bf74f4c56af6","Type":"ContainerDied","Data":"2d73fde4b9f2ec92f960a2cfcecf513fa043a322dad6095816dbe7c008eb8150"}
Sep 15 21:40:06.744380 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.744268    2578 scope.go:117] "RemoveContainer" containerID="be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"
Sep 15 21:40:06.757162 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.757142    2578 scope.go:117] "RemoveContainer" containerID="be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"
Sep 15 21:40:06.757473 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:40:06.757454    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0\": container with ID starting with be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0 not found: ID does not exist" containerID="be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"
Sep 15 21:40:06.757520 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.757483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0"} err="failed to get container status \"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0\": rpc error: code = NotFound desc = could not find container \"be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0\": container with ID starting with be591cc954f16763ef1c81d51043278e464801fcc2546ca5b2cfb333f0fc91f0 not found: ID does not exist"
Sep 15 21:40:06.765301 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.765270    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:40:06.771405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:06.771379    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac4411/e2e-conflict-a-473af5c1-kserve-7777f9dcf7-xn8hs"]
Sep 15 21:40:08.076766 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:40:08.076707    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1859a80-4fe5-4230-a057-bf74f4c56af6" path="/var/lib/kubelet/pods/b1859a80-4fe5-4230-a057-bf74f4c56af6/volumes"
Sep 15 21:41:28.084275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.084239    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:41:28.085019 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.084997    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" containerName="payload-pre-processing"
Sep 15 21:41:28.085124 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.085022    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" containerName="payload-pre-processing"
Sep 15 21:41:28.085124 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.085050    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1859a80-4fe5-4230-a057-bf74f4c56af6" containerName="main"
Sep 15 21:41:28.085124 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.085058    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1859a80-4fe5-4230-a057-bf74f4c56af6" containerName="main"
Sep 15 21:41:28.085283 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.085166    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1859a80-4fe5-4230-a057-bf74f4c56af6" containerName="main"
Sep 15 21:41:28.085283 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.085186    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6988b56d-91a0-430d-94b7-9b1fbe763ee1" containerName="payload-pre-processing"
Sep 15 21:41:28.088649 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.088628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.090533 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.090493    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b0aa2e\""
Sep 15 21:41:28.090906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.090651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b0aa2e-dockercfg-xqvmg\""
Sep 15 21:41:28.098898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.098879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:41:28.101398 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.101378    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:41:28.105343 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.105323    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.112742 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.112698    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:41:28.186529 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.186495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.186768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.186611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77klr\" (UniqueName: \"kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.186768 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.186675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.186902 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.186799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdh56\" (UniqueName: \"kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.287925 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.287890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.288115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.287949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdh56\" (UniqueName: \"kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.288115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.287992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.288115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.288052    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77klr\" (UniqueName: \"kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.288609 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.288582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.288864 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.288841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.295228 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.295205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77klr\" (UniqueName: \"kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr\") pod \"payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") " pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.295332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.295275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdh56\" (UniqueName: \"kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56\") pod \"payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.400088 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.399999    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:28.417557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.417524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:28.695557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.695474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:41:28.698566 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:41:28.698517    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc230cea2_7fc4_47f8_aff2_5e21ff286046.slice/crio-57f2b556788601a4aa713175af36f4bc10ef08cc962189fc248fba8bc73b2934 WatchSource:0}: Error finding container 57f2b556788601a4aa713175af36f4bc10ef08cc962189fc248fba8bc73b2934: Status 404 returned error can't find the container with id 57f2b556788601a4aa713175af36f4bc10ef08cc962189fc248fba8bc73b2934
Sep 15 21:41:28.719198 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:28.719175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:41:28.720965 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:41:28.720942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f391135_3881_4374_a4dc_ba1be26a6021.slice/crio-561340a68f96e796053500ebd9145f00a8d3ae52aa4fa77118e851ec6c5e41d4 WatchSource:0}: Error finding container 561340a68f96e796053500ebd9145f00a8d3ae52aa4fa77118e851ec6c5e41d4: Status 404 returned error can't find the container with id 561340a68f96e796053500ebd9145f00a8d3ae52aa4fa77118e851ec6c5e41d4
Sep 15 21:41:29.200479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.200295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" event={"ID":"0f391135-3881-4374-a4dc-ba1be26a6021","Type":"ContainerStarted","Data":"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"}
Sep 15 21:41:29.200479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.200370    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:29.200479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.200398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" event={"ID":"0f391135-3881-4374-a4dc-ba1be26a6021","Type":"ContainerStarted","Data":"561340a68f96e796053500ebd9145f00a8d3ae52aa4fa77118e851ec6c5e41d4"}
Sep 15 21:41:29.202133 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.202108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" event={"ID":"c230cea2-7fc4-47f8-aff2-5e21ff286046","Type":"ContainerStarted","Data":"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"}
Sep 15 21:41:29.202276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.202155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" event={"ID":"c230cea2-7fc4-47f8-aff2-5e21ff286046","Type":"ContainerStarted","Data":"57f2b556788601a4aa713175af36f4bc10ef08cc962189fc248fba8bc73b2934"}
Sep 15 21:41:29.202276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.202198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:29.217079 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.217017    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" podStartSLOduration=1.216994974 podStartE2EDuration="1.216994974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:29.215654653 +0000 UTC m=+1517.616627901" watchObservedRunningTime="2026-09-15 21:41:29.216994974 +0000 UTC m=+1517.617968222"
Sep 15 21:41:29.232871 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:29.232820    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" podStartSLOduration=1.232803828 podStartE2EDuration="1.232803828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:29.230654634 +0000 UTC m=+1517.631627881" watchObservedRunningTime="2026-09-15 21:41:29.232803828 +0000 UTC m=+1517.633777074"
Sep 15 21:41:40.209809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:40.209770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:41:40.212210 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:40.210050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:41:53.115991 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.115954    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:41:53.120056 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.120033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.121994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.121970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-57a27b-openshift-default-dockercfg-ctzlm\""
Sep 15 21:41:53.122139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.121972    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:41:53.131623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.131586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:41:53.255378 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75bk\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255707 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255797 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255907 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.255965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.255897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75bk\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.357867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.357868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.358259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.358229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.358449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.358416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.358449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.358328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.358600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.358379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.358600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.358523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.360162 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.360138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.360497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.360477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.367129 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.367072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.367302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.367282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75bk\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk\") pod \"e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") " pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.433457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.433422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:53.718383 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.718352    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:41:53.720222 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:41:53.720188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dae7f9_0749_4b74_895d_d58402d943be.slice/crio-05fac3a53d5345e254c8c268f55890d787e001c68553ab8f157d85829f966c52 WatchSource:0}: Error finding container 05fac3a53d5345e254c8c268f55890d787e001c68553ab8f157d85829f966c52: Status 404 returned error can't find the container with id 05fac3a53d5345e254c8c268f55890d787e001c68553ab8f157d85829f966c52
Sep 15 21:41:53.723321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.723274    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:41:53.723404 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.723374    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:41:53.723460 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:53.723444    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:41:54.348486 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:54.348448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerStarted","Data":"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"}
Sep 15 21:41:54.348918 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:54.348503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerStarted","Data":"05fac3a53d5345e254c8c268f55890d787e001c68553ab8f157d85829f966c52"}
Sep 15 21:41:54.370354 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:54.370296    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podStartSLOduration=1.370277162 podStartE2EDuration="1.370277162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:54.366652519 +0000 UTC m=+1542.767625756" watchObservedRunningTime="2026-09-15 21:41:54.370277162 +0000 UTC m=+1542.771250410"
Sep 15 21:41:54.434008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:54.433962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:54.439623 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:54.439593    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:55.354314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:55.354276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:41:55.355107 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:41:55.355085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:39.614734 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.614676    2578 generic.go:356] "Generic (PLEG): container finished" podID="c2dae7f9-0749-4b74-895d-d58402d943be" containerID="ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722" exitCode=0
Sep 15 21:42:39.615169 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.614757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerDied","Data":"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"}
Sep 15 21:42:39.615284 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.615269    2578 scope.go:117] "RemoveContainer" containerID="ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"
Sep 15 21:42:39.616684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.616656    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:42:39.616758 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.616740    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:42:39.616844 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:39.616785    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:42:40.355175 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:40.355137    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:40.622861 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:40.622766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerStarted","Data":"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"}
Sep 15 21:42:41.434314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:41.434272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:41.435537 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:41.435511    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:41.435600 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:41.435570    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:42.434048 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:42.434008    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:42.434472 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:42.434084    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:43.433832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:43.433796    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:43.434015 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:43.433854    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:44.434789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:44.434752    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:44.435268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:44.434808    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:45.434592 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:45.434553    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:45.434814 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:45.434625    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:45.995273 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:45.995212    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:42:45.995488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:45.995463    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" podUID="0f391135-3881-4374-a4dc-ba1be26a6021" containerName="payload-processing" containerID="cri-o://4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd" gracePeriod=30
Sep 15 21:42:46.011890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.011856    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:42:46.012123 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.012089    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" podUID="c230cea2-7fc4-47f8-aff2-5e21ff286046" containerName="payload-pre-processing" containerID="cri-o://4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c" gracePeriod=30
Sep 15 21:42:46.304453 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.304387    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:42:46.344959 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.344912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume\") pod \"0f391135-3881-4374-a4dc-ba1be26a6021\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") "
Sep 15 21:42:46.345169 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.344997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77klr\" (UniqueName: \"kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr\") pod \"0f391135-3881-4374-a4dc-ba1be26a6021\" (UID: \"0f391135-3881-4374-a4dc-ba1be26a6021\") "
Sep 15 21:42:46.345371 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.345343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f391135-3881-4374-a4dc-ba1be26a6021" (UID: "0f391135-3881-4374-a4dc-ba1be26a6021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:46.345568 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.345542    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f391135-3881-4374-a4dc-ba1be26a6021-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:42:46.347683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.347654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr" (OuterVolumeSpecName: "kube-api-access-77klr") pod "0f391135-3881-4374-a4dc-ba1be26a6021" (UID: "0f391135-3881-4374-a4dc-ba1be26a6021"). InnerVolumeSpecName "kube-api-access-77klr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:46.363235 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.363211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:42:46.433857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.433818    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 15 21:42:46.434075 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.433887    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 15 21:42:46.446347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.446295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume\") pod \"c230cea2-7fc4-47f8-aff2-5e21ff286046\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") "
Sep 15 21:42:46.446347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.446345    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdh56\" (UniqueName: \"kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56\") pod \"c230cea2-7fc4-47f8-aff2-5e21ff286046\" (UID: \"c230cea2-7fc4-47f8-aff2-5e21ff286046\") "
Sep 15 21:42:46.446922 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.446826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c230cea2-7fc4-47f8-aff2-5e21ff286046" (UID: "c230cea2-7fc4-47f8-aff2-5e21ff286046"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:46.446922 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.446889    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77klr\" (UniqueName: \"kubernetes.io/projected/0f391135-3881-4374-a4dc-ba1be26a6021-kube-api-access-77klr\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:42:46.448646 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.448615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56" (OuterVolumeSpecName: "kube-api-access-hdh56") pod "c230cea2-7fc4-47f8-aff2-5e21ff286046" (UID: "c230cea2-7fc4-47f8-aff2-5e21ff286046"). InnerVolumeSpecName "kube-api-access-hdh56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:46.547763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.547646    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c230cea2-7fc4-47f8-aff2-5e21ff286046-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:42:46.547763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.547680    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdh56\" (UniqueName: \"kubernetes.io/projected/c230cea2-7fc4-47f8-aff2-5e21ff286046-kube-api-access-hdh56\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:42:46.665939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.665898    2578 generic.go:356] "Generic (PLEG): container finished" podID="c230cea2-7fc4-47f8-aff2-5e21ff286046" containerID="4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c" exitCode=0
Sep 15 21:42:46.666117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.665946    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"
Sep 15 21:42:46.666117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.665987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" event={"ID":"c230cea2-7fc4-47f8-aff2-5e21ff286046","Type":"ContainerDied","Data":"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"}
Sep 15 21:42:46.666117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.666045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6" event={"ID":"c230cea2-7fc4-47f8-aff2-5e21ff286046","Type":"ContainerDied","Data":"57f2b556788601a4aa713175af36f4bc10ef08cc962189fc248fba8bc73b2934"}
Sep 15 21:42:46.666117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.666066    2578 scope.go:117] "RemoveContainer" containerID="4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"
Sep 15 21:42:46.667860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.667564    2578 generic.go:356] "Generic (PLEG): container finished" podID="0f391135-3881-4374-a4dc-ba1be26a6021" containerID="4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd" exitCode=0
Sep 15 21:42:46.667860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.667596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" event={"ID":"0f391135-3881-4374-a4dc-ba1be26a6021","Type":"ContainerDied","Data":"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"}
Sep 15 21:42:46.667860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.667617    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"
Sep 15 21:42:46.667860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.667635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv" event={"ID":"0f391135-3881-4374-a4dc-ba1be26a6021","Type":"ContainerDied","Data":"561340a68f96e796053500ebd9145f00a8d3ae52aa4fa77118e851ec6c5e41d4"}
Sep 15 21:42:46.681776 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.681239    2578 scope.go:117] "RemoveContainer" containerID="4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"
Sep 15 21:42:46.681888 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:42:46.681846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c\": container with ID starting with 4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c not found: ID does not exist" containerID="4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"
Sep 15 21:42:46.681952 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.681887    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c"} err="failed to get container status \"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c\": rpc error: code = NotFound desc = could not find container \"4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c\": container with ID starting with 4a300d19bd2223fbd22e44c5ae667c7ca5f2ddf4400b4bf6a1cdcebf56ecb50c not found: ID does not exist"
Sep 15 21:42:46.682026 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.681951    2578 scope.go:117] "RemoveContainer" containerID="4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"
Sep 15 21:42:46.691760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.691733    2578 scope.go:117] "RemoveContainer" containerID="4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"
Sep 15 21:42:46.692076 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:42:46.692056    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd\": container with ID starting with 4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd not found: ID does not exist" containerID="4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"
Sep 15 21:42:46.692131 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.692086    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd"} err="failed to get container status \"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd\": rpc error: code = NotFound desc = could not find container \"4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd\": container with ID starting with 4a3304fd3f7b8eb312a840dc780bc3350db3830fc0b157271ed82dade32d3edd not found: ID does not exist"
Sep 15 21:42:46.697792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.697762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:42:46.706132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.706103    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b0aa2e-6fdb4849f4-s9br6"]
Sep 15 21:42:46.723127 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.723097    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:42:46.735730 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:46.735684    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b0aa2e-84f8b8f469-qcsbv"]
Sep 15 21:42:47.438130 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:47.438046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:47.438335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:47.438314    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:47.439081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:47.439057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:42:48.076526 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:48.076492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f391135-3881-4374-a4dc-ba1be26a6021" path="/var/lib/kubelet/pods/0f391135-3881-4374-a4dc-ba1be26a6021/volumes"
Sep 15 21:42:48.076975 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:42:48.076955    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c230cea2-7fc4-47f8-aff2-5e21ff286046" path="/var/lib/kubelet/pods/c230cea2-7fc4-47f8-aff2-5e21ff286046/volumes"
Sep 15 21:43:16.081752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:16.081680    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:43:16.082609 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:16.082549    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" containerID="cri-o://26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96" gracePeriod=30
Sep 15 21:43:17.440789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:17.440731    2578 patch_prober.go:28] interesting pod/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:43:17.441197 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:17.440829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:43:21.348420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.348350    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:43:21.432156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432156 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432243    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l75bk\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432410    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:21.432628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:21.432628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:21.432948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data\") pod \"c2dae7f9-0749-4b74-895d-d58402d943be\" (UID: \"c2dae7f9-0749-4b74-895d-d58402d943be\") "
Sep 15 21:43:21.432948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432845    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:21.433061 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.432965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:21.433211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.433188    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.433279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.433214    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-data\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.433279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.433223    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2dae7f9-0749-4b74-895d-d58402d943be-istiod-ca-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.433279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.433231    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-workload-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.433279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.433239    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-credential-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.434843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.434814    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:21.434991 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.434861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:21.435103 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.435084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk" (OuterVolumeSpecName: "kube-api-access-l75bk") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "kube-api-access-l75bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:21.435103 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.435092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2dae7f9-0749-4b74-895d-d58402d943be" (UID: "c2dae7f9-0749-4b74-895d-d58402d943be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:43:21.534678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.534630    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2dae7f9-0749-4b74-895d-d58402d943be-istio-podinfo\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.534678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.534667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75bk\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-kube-api-access-l75bk\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.534678 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.534682    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2dae7f9-0749-4b74-895d-d58402d943be-istio-token\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.534965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.534695    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2dae7f9-0749-4b74-895d-d58402d943be-istio-envoy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:21.871826 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.871784    2578 generic.go:356] "Generic (PLEG): container finished" podID="c2dae7f9-0749-4b74-895d-d58402d943be" containerID="26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96" exitCode=0
Sep 15 21:43:21.871826 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.871825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"
Sep 15 21:43:21.872074 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.871861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerDied","Data":"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"}
Sep 15 21:43:21.872074 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.871914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml" event={"ID":"c2dae7f9-0749-4b74-895d-d58402d943be","Type":"ContainerDied","Data":"05fac3a53d5345e254c8c268f55890d787e001c68553ab8f157d85829f966c52"}
Sep 15 21:43:21.872074 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.871936    2578 scope.go:117] "RemoveContainer" containerID="26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"
Sep 15 21:43:21.886050 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.886025    2578 scope.go:117] "RemoveContainer" containerID="ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"
Sep 15 21:43:21.895541 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.895519    2578 scope.go:117] "RemoveContainer" containerID="26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"
Sep 15 21:43:21.895898 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:21.895875    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96\": container with ID starting with 26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96 not found: ID does not exist" containerID="26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"
Sep 15 21:43:21.895963 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.895909    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96"} err="failed to get container status \"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96\": rpc error: code = NotFound desc = could not find container \"26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96\": container with ID starting with 26ca074441ff343a1851c2531d61f2367c948e5cd180c348f845dead1cd64a96 not found: ID does not exist"
Sep 15 21:43:21.895963 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.895926    2578 scope.go:117] "RemoveContainer" containerID="ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"
Sep 15 21:43:21.896176 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:21.896154    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722\": container with ID starting with ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722 not found: ID does not exist" containerID="ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"
Sep 15 21:43:21.896260 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.896186    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722"} err="failed to get container status \"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722\": rpc error: code = NotFound desc = could not find container \"ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722\": container with ID starting with ddd20883c39d4416eceadad21a70ddebc50c11ead77084379bfa98cab894d722 not found: ID does not exist"
Sep 15 21:43:21.903045 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.903022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:43:21.909927 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:21.909901    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-57a27b-openshift-default-6dd4dd6858-qzdml"]
Sep 15 21:43:22.078332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:22.078298    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" path="/var/lib/kubelet/pods/c2dae7f9-0749-4b74-895d-d58402d943be/volumes"
Sep 15 21:43:26.989753 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:26.989704    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:43:26.990255 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:26.989922    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" podUID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" containerName="payload-processing" containerID="cri-o://90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76" gracePeriod=30
Sep 15 21:43:27.253864 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.253795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:43:27.398252 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.398216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume\") pod \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") "
Sep 15 21:43:27.398444 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.398318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smr94\" (UniqueName: \"kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94\") pod \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\" (UID: \"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771\") "
Sep 15 21:43:27.398673 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.398647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" (UID: "9e54cd9b-7a2e-43c4-b650-c5e7ae51a771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:27.398817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.398800    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:27.400395 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.400374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94" (OuterVolumeSpecName: "kube-api-access-smr94") pod "9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" (UID: "9e54cd9b-7a2e-43c4-b650-c5e7ae51a771"). InnerVolumeSpecName "kube-api-access-smr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:27.508650 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.508547    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smr94\" (UniqueName: \"kubernetes.io/projected/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771-kube-api-access-smr94\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:27.907935 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.907837    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" containerID="90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76" exitCode=0
Sep 15 21:43:27.907935 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.907900    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"
Sep 15 21:43:27.907935 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.907926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" event={"ID":"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771","Type":"ContainerDied","Data":"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"}
Sep 15 21:43:27.908191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.908001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt" event={"ID":"9e54cd9b-7a2e-43c4-b650-c5e7ae51a771","Type":"ContainerDied","Data":"7fef5b864c50c27172f568b55283725bd58eb8df5730f21d1510900cd3ca880c"}
Sep 15 21:43:27.908191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.908026    2578 scope.go:117] "RemoveContainer" containerID="90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"
Sep 15 21:43:27.917996 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.917971    2578 scope.go:117] "RemoveContainer" containerID="90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"
Sep 15 21:43:27.918285 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:27.918266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76\": container with ID starting with 90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76 not found: ID does not exist" containerID="90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"
Sep 15 21:43:27.918334 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.918295    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76"} err="failed to get container status \"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76\": rpc error: code = NotFound desc = could not find container \"90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76\": container with ID starting with 90cf5aef02d8b6a6db11a7ef812c49857a3bf12b8a59fba6486efdaccd331b76 not found: ID does not exist"
Sep 15 21:43:27.931931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.931900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:43:27.940336 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:27.940304    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2d4fab-686dd75bdc-pf9rt"]
Sep 15 21:43:28.076760 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:28.076700    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" path="/var/lib/kubelet/pods/9e54cd9b-7a2e-43c4-b650-c5e7ae51a771/volumes"
Sep 15 21:43:41.502115 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:41.502078    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:43:41.506971 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:41.506940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:43:44.940690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:44.940652    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:43:44.941215 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:44.940940    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" podUID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" containerName="maas-api" containerID="cri-o://ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc" gracePeriod=30
Sep 15 21:43:44.995435 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:44.995398    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:43:44.995727 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:44.995661    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" podUID="fffc881e-e85e-4250-b7c7-69230edda6b8" containerName="payload-processing" containerID="cri-o://f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88" gracePeriod=30
Sep 15 21:43:45.004377 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.004348    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:43:45.004577 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.004549    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" podUID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" containerName="payload-pre-processing" containerID="cri-o://5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e" gracePeriod=30
Sep 15 21:43:45.281934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.281902    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:43:45.309110 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.309078    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:43:45.321480 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.321457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:43:45.404352 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n5bl\" (UniqueName: \"kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl\") pod \"fffc881e-e85e-4250-b7c7-69230edda6b8\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") "
Sep 15 21:43:45.404581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") pod \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") "
Sep 15 21:43:45.404581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume\") pod \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") "
Sep 15 21:43:45.404581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwrz5\" (UniqueName: \"kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5\") pod \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") "
Sep 15 21:43:45.404581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca\") pod \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") "
Sep 15 21:43:45.404581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjnb9\" (UniqueName: \"kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9\") pod \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\" (UID: \"f5bd7051-8a23-4306-a4cf-40d95ac44f50\") "
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume\") pod \"fffc881e-e85e-4250-b7c7-69230edda6b8\" (UID: \"fffc881e-e85e-4250-b7c7-69230edda6b8\") "
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs\") pod \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\" (UID: \"9233d557-a6aa-4d6a-aecd-58cab5f517cd\") "
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.404972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5bd7051-8a23-4306-a4cf-40d95ac44f50" (UID: "f5bd7051-8a23-4306-a4cf-40d95ac44f50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.405032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fffc881e-e85e-4250-b7c7-69230edda6b8" (UID: "fffc881e-e85e-4250-b7c7-69230edda6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.405231    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fffc881e-e85e-4250-b7c7-69230edda6b8-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.405255    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bd7051-8a23-4306-a4cf-40d95ac44f50-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.405681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9233d557-a6aa-4d6a-aecd-58cab5f517cd" (UID: "9233d557-a6aa-4d6a-aecd-58cab5f517cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:45.406910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.406745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9233d557-a6aa-4d6a-aecd-58cab5f517cd" (UID: "9233d557-a6aa-4d6a-aecd-58cab5f517cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:45.407274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.407115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl" (OuterVolumeSpecName: "kube-api-access-5n5bl") pod "fffc881e-e85e-4250-b7c7-69230edda6b8" (UID: "fffc881e-e85e-4250-b7c7-69230edda6b8"). InnerVolumeSpecName "kube-api-access-5n5bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:45.407274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.407188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9" (OuterVolumeSpecName: "kube-api-access-zjnb9") pod "f5bd7051-8a23-4306-a4cf-40d95ac44f50" (UID: "f5bd7051-8a23-4306-a4cf-40d95ac44f50"). InnerVolumeSpecName "kube-api-access-zjnb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:45.407274 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.407202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5" (OuterVolumeSpecName: "kube-api-access-vwrz5") pod "9233d557-a6aa-4d6a-aecd-58cab5f517cd" (UID: "9233d557-a6aa-4d6a-aecd-58cab5f517cd"). InnerVolumeSpecName "kube-api-access-vwrz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:45.407555 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.407537    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9233d557-a6aa-4d6a-aecd-58cab5f517cd" (UID: "9233d557-a6aa-4d6a-aecd-58cab5f517cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506613    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwrz5\" (UniqueName: \"kubernetes.io/projected/9233d557-a6aa-4d6a-aecd-58cab5f517cd-kube-api-access-vwrz5\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506645    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9233d557-a6aa-4d6a-aecd-58cab5f517cd-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506655    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnb9\" (UniqueName: \"kubernetes.io/projected/f5bd7051-8a23-4306-a4cf-40d95ac44f50-kube-api-access-zjnb9\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506668    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506677    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n5bl\" (UniqueName: \"kubernetes.io/projected/fffc881e-e85e-4250-b7c7-69230edda6b8-kube-api-access-5n5bl\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:45.506697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:45.506685    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9233d557-a6aa-4d6a-aecd-58cab5f517cd-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:46.019400 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.019357    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" containerID="5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e" exitCode=0
Sep 15 21:43:46.019873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.019417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"
Sep 15 21:43:46.019873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.019437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" event={"ID":"f5bd7051-8a23-4306-a4cf-40d95ac44f50","Type":"ContainerDied","Data":"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"}
Sep 15 21:43:46.019873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.019515    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks" event={"ID":"f5bd7051-8a23-4306-a4cf-40d95ac44f50","Type":"ContainerDied","Data":"02a739cfeb0c89837b405cb788d3ee24f00eec70d2fc650cb24de52a5dbd7c28"}
Sep 15 21:43:46.019873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.019538    2578 scope.go:117] "RemoveContainer" containerID="5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"
Sep 15 21:43:46.021088 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.021063    2578 generic.go:356] "Generic (PLEG): container finished" podID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" containerID="ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc" exitCode=0
Sep 15 21:43:46.021217 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.021133    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"
Sep 15 21:43:46.021217 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.021181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" event={"ID":"9233d557-a6aa-4d6a-aecd-58cab5f517cd","Type":"ContainerDied","Data":"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"}
Sep 15 21:43:46.021347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.021234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn" event={"ID":"9233d557-a6aa-4d6a-aecd-58cab5f517cd","Type":"ContainerDied","Data":"25ed5fca5355c1a2cd314beca77d35205ec3ad87d9b6a3b3889bc20e8294777a"}
Sep 15 21:43:46.022797 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.022770    2578 generic.go:356] "Generic (PLEG): container finished" podID="fffc881e-e85e-4250-b7c7-69230edda6b8" containerID="f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88" exitCode=0
Sep 15 21:43:46.022934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.022839    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"
Sep 15 21:43:46.022934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.022848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" event={"ID":"fffc881e-e85e-4250-b7c7-69230edda6b8","Type":"ContainerDied","Data":"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"}
Sep 15 21:43:46.022934 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.022899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh" event={"ID":"fffc881e-e85e-4250-b7c7-69230edda6b8","Type":"ContainerDied","Data":"49f8ca36a01922c4221fccc9db9555c55de302ecd9bfdab8751974c35e3d7be0"}
Sep 15 21:43:46.029783 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.029670    2578 scope.go:117] "RemoveContainer" containerID="5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"
Sep 15 21:43:46.030139 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:46.030041    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e\": container with ID starting with 5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e not found: ID does not exist" containerID="5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"
Sep 15 21:43:46.030139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.030077    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e"} err="failed to get container status \"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e\": rpc error: code = NotFound desc = could not find container \"5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e\": container with ID starting with 5e90049d216fac5f9a99ddfdf865a616ec1c1a844fee5e62f1aa8acac00c1e9e not found: ID does not exist"
Sep 15 21:43:46.030139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.030101    2578 scope.go:117] "RemoveContainer" containerID="ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"
Sep 15 21:43:46.039177 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.039156    2578 scope.go:117] "RemoveContainer" containerID="ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"
Sep 15 21:43:46.039452 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:46.039434    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc\": container with ID starting with ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc not found: ID does not exist" containerID="ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"
Sep 15 21:43:46.039502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.039459    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc"} err="failed to get container status \"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc\": rpc error: code = NotFound desc = could not find container \"ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc\": container with ID starting with ca18987a709dd60d546f97bbdd95d5cafb5e386d2e8ba143d92a64dabeedf5dc not found: ID does not exist"
Sep 15 21:43:46.039502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.039474    2578 scope.go:117] "RemoveContainer" containerID="f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"
Sep 15 21:43:46.048502 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.048481    2578 scope.go:117] "RemoveContainer" containerID="f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"
Sep 15 21:43:46.048845 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:46.048821    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88\": container with ID starting with f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88 not found: ID does not exist" containerID="f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"
Sep 15 21:43:46.048910 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.048855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88"} err="failed to get container status \"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88\": rpc error: code = NotFound desc = could not find container \"f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88\": container with ID starting with f2f629f037c4a083d4132ffce8058fb1316749a75f15e715b2eed0b288593e88 not found: ID does not exist"
Sep 15 21:43:46.049046 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.049032    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:43:46.052582 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.052558    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fb0f0e-57c44c7b9d-vnzks"]
Sep 15 21:43:46.061974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.061951    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:43:46.066593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.066561    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fb0f0e-7bdd5748b5-4nzvh"]
Sep 15 21:43:46.077421 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.077393    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" path="/var/lib/kubelet/pods/f5bd7051-8a23-4306-a4cf-40d95ac44f50/volumes"
Sep 15 21:43:46.077857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.077842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffc881e-e85e-4250-b7c7-69230edda6b8" path="/var/lib/kubelet/pods/fffc881e-e85e-4250-b7c7-69230edda6b8/volumes"
Sep 15 21:43:46.078159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.078120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:43:46.080746 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:46.080708    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fb0f0e-5d78d89699-6tqfn"]
Sep 15 21:43:48.077234 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:48.077201    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" path="/var/lib/kubelet/pods/9233d557-a6aa-4d6a-aecd-58cab5f517cd/volumes"
Sep 15 21:43:52.023095 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.023059    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:43:52.023519 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.023292    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" podUID="356859e8-5908-4d4b-a2bc-8a9c204d7418" containerName="maas-api" containerID="cri-o://f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d" gracePeriod=30
Sep 15 21:43:52.117356 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.117319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:43:52.117680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.117638    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" podUID="69d6935b-0628-428a-909d-832e3d53d826" containerName="payload-pre-processing" containerID="cri-o://8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da" gracePeriod=30
Sep 15 21:43:52.295496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.295430    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:43:52.376482 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.376453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jcmh\" (UniqueName: \"kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh\") pod \"356859e8-5908-4d4b-a2bc-8a9c204d7418\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") "
Sep 15 21:43:52.376660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.376531    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs\") pod \"356859e8-5908-4d4b-a2bc-8a9c204d7418\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") "
Sep 15 21:43:52.376660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.376585    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca\") pod \"356859e8-5908-4d4b-a2bc-8a9c204d7418\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") "
Sep 15 21:43:52.376660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.376630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") pod \"356859e8-5908-4d4b-a2bc-8a9c204d7418\" (UID: \"356859e8-5908-4d4b-a2bc-8a9c204d7418\") "
Sep 15 21:43:52.377049 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.377021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "356859e8-5908-4d4b-a2bc-8a9c204d7418" (UID: "356859e8-5908-4d4b-a2bc-8a9c204d7418"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:52.377994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.377962    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/356859e8-5908-4d4b-a2bc-8a9c204d7418-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.379231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.379189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh" (OuterVolumeSpecName: "kube-api-access-5jcmh") pod "356859e8-5908-4d4b-a2bc-8a9c204d7418" (UID: "356859e8-5908-4d4b-a2bc-8a9c204d7418"). InnerVolumeSpecName "kube-api-access-5jcmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:52.379489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.379461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "356859e8-5908-4d4b-a2bc-8a9c204d7418" (UID: "356859e8-5908-4d4b-a2bc-8a9c204d7418"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:52.380411 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.380389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "356859e8-5908-4d4b-a2bc-8a9c204d7418" (UID: "356859e8-5908-4d4b-a2bc-8a9c204d7418"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:52.391084 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.391052    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:43:52.478397 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478362    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume\") pod \"69d6935b-0628-428a-909d-832e3d53d826\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") "
Sep 15 21:43:52.478607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478431    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nhxm\" (UniqueName: \"kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm\") pod \"69d6935b-0628-428a-909d-832e3d53d826\" (UID: \"69d6935b-0628-428a-909d-832e3d53d826\") "
Sep 15 21:43:52.478788 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478767    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.478876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478788    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jcmh\" (UniqueName: \"kubernetes.io/projected/356859e8-5908-4d4b-a2bc-8a9c204d7418-kube-api-access-5jcmh\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.478876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478803    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/356859e8-5908-4d4b-a2bc-8a9c204d7418-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.478876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.478833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69d6935b-0628-428a-909d-832e3d53d826" (UID: "69d6935b-0628-428a-909d-832e3d53d826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:52.480898 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.480872    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm" (OuterVolumeSpecName: "kube-api-access-6nhxm") pod "69d6935b-0628-428a-909d-832e3d53d826" (UID: "69d6935b-0628-428a-909d-832e3d53d826"). InnerVolumeSpecName "kube-api-access-6nhxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:52.579904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.579818    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d6935b-0628-428a-909d-832e3d53d826-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.579904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.579847    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhxm\" (UniqueName: \"kubernetes.io/projected/69d6935b-0628-428a-909d-832e3d53d826-kube-api-access-6nhxm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.700475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.700430    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:43:52.700743 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.700693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" podUID="4efb254f-4201-473e-ac54-62485172d790" containerName="maas-api" containerID="cri-o://15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029" gracePeriod=30
Sep 15 21:43:52.940321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.940283    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:43:52.988590 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.988557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpcs4\" (UniqueName: \"kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4\") pod \"4efb254f-4201-473e-ac54-62485172d790\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") "
Sep 15 21:43:52.988816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.988631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs\") pod \"4efb254f-4201-473e-ac54-62485172d790\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") "
Sep 15 21:43:52.988816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.988678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca\") pod \"4efb254f-4201-473e-ac54-62485172d790\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") "
Sep 15 21:43:52.988816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.988784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls\") pod \"4efb254f-4201-473e-ac54-62485172d790\" (UID: \"4efb254f-4201-473e-ac54-62485172d790\") "
Sep 15 21:43:52.989090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.989053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4efb254f-4201-473e-ac54-62485172d790" (UID: "4efb254f-4201-473e-ac54-62485172d790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:43:52.989238 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.989183    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4efb254f-4201-473e-ac54-62485172d790-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:52.990790 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.990762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4" (OuterVolumeSpecName: "kube-api-access-lpcs4") pod "4efb254f-4201-473e-ac54-62485172d790" (UID: "4efb254f-4201-473e-ac54-62485172d790"). InnerVolumeSpecName "kube-api-access-lpcs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:52.990889 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.990809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4efb254f-4201-473e-ac54-62485172d790" (UID: "4efb254f-4201-473e-ac54-62485172d790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:52.990889 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:52.990823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4efb254f-4201-473e-ac54-62485172d790" (UID: "4efb254f-4201-473e-ac54-62485172d790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:53.070360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.070319    2578 generic.go:356] "Generic (PLEG): container finished" podID="4efb254f-4201-473e-ac54-62485172d790" containerID="15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029" exitCode=0
Sep 15 21:43:53.070360 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.070360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"
Sep 15 21:43:53.070884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.070412    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" event={"ID":"4efb254f-4201-473e-ac54-62485172d790","Type":"ContainerDied","Data":"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"}
Sep 15 21:43:53.070884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.070475    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" event={"ID":"4efb254f-4201-473e-ac54-62485172d790","Type":"ContainerDied","Data":"9e3fe23b729ddc2936fd83e18989e63048301100b77be8748cb15b0263117cb9"}
Sep 15 21:43:53.070884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.070507    2578 scope.go:117] "RemoveContainer" containerID="15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"
Sep 15 21:43:53.072312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.072054    2578 generic.go:356] "Generic (PLEG): container finished" podID="356859e8-5908-4d4b-a2bc-8a9c204d7418" containerID="f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d" exitCode=0
Sep 15 21:43:53.072312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.072105    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"
Sep 15 21:43:53.072312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.072155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" event={"ID":"356859e8-5908-4d4b-a2bc-8a9c204d7418","Type":"ContainerDied","Data":"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"}
Sep 15 21:43:53.072312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.072193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj" event={"ID":"356859e8-5908-4d4b-a2bc-8a9c204d7418","Type":"ContainerDied","Data":"8a94880b5fdedb8d29b0d66949e8623ec8d03201af3b960c45c8cbea014fa30f"}
Sep 15 21:43:53.073941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.073886    2578 generic.go:356] "Generic (PLEG): container finished" podID="69d6935b-0628-428a-909d-832e3d53d826" containerID="8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da" exitCode=0
Sep 15 21:43:53.073941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.073931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" event={"ID":"69d6935b-0628-428a-909d-832e3d53d826","Type":"ContainerDied","Data":"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"}
Sep 15 21:43:53.074134 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.073953    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"
Sep 15 21:43:53.074134 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.073971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh" event={"ID":"69d6935b-0628-428a-909d-832e3d53d826","Type":"ContainerDied","Data":"bbfd59823292ff5deaee1ad6699f0557ea87b42ee9316806214ad8cf69657b00"}
Sep 15 21:43:53.084218 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.083061    2578 scope.go:117] "RemoveContainer" containerID="15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"
Sep 15 21:43:53.084218 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:53.083574    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029\": container with ID starting with 15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029 not found: ID does not exist" containerID="15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"
Sep 15 21:43:53.084218 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.083608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029"} err="failed to get container status \"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029\": rpc error: code = NotFound desc = could not find container \"15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029\": container with ID starting with 15d126ec0a740e241de83c1552d6e1cbdb8c5e760e4f1b8a39eccbd8f5525029 not found: ID does not exist"
Sep 15 21:43:53.084218 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.083632    2578 scope.go:117] "RemoveContainer" containerID="f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"
Sep 15 21:43:53.090191 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.090162    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpcs4\" (UniqueName: \"kubernetes.io/projected/4efb254f-4201-473e-ac54-62485172d790-kube-api-access-lpcs4\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:53.090296 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.090191    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:53.090296 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.090207    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4efb254f-4201-473e-ac54-62485172d790-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:53.093612 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.093593    2578 scope.go:117] "RemoveContainer" containerID="f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"
Sep 15 21:43:53.093934 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:53.093915    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d\": container with ID starting with f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d not found: ID does not exist" containerID="f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"
Sep 15 21:43:53.093992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.093942    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d"} err="failed to get container status \"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d\": rpc error: code = NotFound desc = could not find container \"f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d\": container with ID starting with f8dc61d9baf106d7e0819c1bb6acd7a0636712c8cc429c254f9b8b6830fb4a5d not found: ID does not exist"
Sep 15 21:43:53.093992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.093958    2578 scope.go:117] "RemoveContainer" containerID="8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"
Sep 15 21:43:53.102174 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.102155    2578 scope.go:117] "RemoveContainer" containerID="8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"
Sep 15 21:43:53.102424 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:53.102406    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da\": container with ID starting with 8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da not found: ID does not exist" containerID="8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"
Sep 15 21:43:53.102483 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.102437    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da"} err="failed to get container status \"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da\": rpc error: code = NotFound desc = could not find container \"8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da\": container with ID starting with 8f7e99585c9aba72880f0ceb3e827cbcf570908609cfc213b47a7c503534e1da not found: ID does not exist"
Sep 15 21:43:53.115066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.115036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:43:53.122294 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.122268    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8"]
Sep 15 21:43:53.137965 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.137872    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:43:53.145051 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.145021    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b99d7b-c9bf8459c-4lrdj"]
Sep 15 21:43:53.162392 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.162357    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:43:53.167705 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:53.167678    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b99d7b-6cc88968cd-nm5mh"]
Sep 15 21:43:54.076783 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:54.076735    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="356859e8-5908-4d4b-a2bc-8a9c204d7418" path="/var/lib/kubelet/pods/356859e8-5908-4d4b-a2bc-8a9c204d7418/volumes"
Sep 15 21:43:54.077254 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:54.077191    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4efb254f-4201-473e-ac54-62485172d790" path="/var/lib/kubelet/pods/4efb254f-4201-473e-ac54-62485172d790/volumes"
Sep 15 21:43:54.077811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:54.077791    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d6935b-0628-428a-909d-832e3d53d826" path="/var/lib/kubelet/pods/69d6935b-0628-428a-909d-832e3d53d826/volumes"
Sep 15 21:43:55.935736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:55.935658    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac4411-5bd5445784-ghgq8" podUID="4efb254f-4201-473e-ac54-62485172d790" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": context deadline exceeded"
Sep 15 21:43:56.522618 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.522582    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" podUID="3514f173-0573-415b-8682-f6c971179d1e" containerName="main" containerID="cri-o://bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b" gracePeriod=45
Sep 15 21:43:56.527243 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.527213    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" podUID="610adc82-acb1-4b98-877d-0bc6eb5e31db" containerName="main" containerID="cri-o://189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625" gracePeriod=45
Sep 15 21:43:56.781939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.781865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:43:56.809092 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.809068    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:43:56.934213 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934415 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrw4n\" (UniqueName: \"kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n\") pod \"3514f173-0573-415b-8682-f6c971179d1e\" (UID: \"3514f173-0573-415b-8682-f6c971179d1e\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb29q\" (UniqueName: \"kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir\") pod \"610adc82-acb1-4b98-877d-0bc6eb5e31db\" (UID: \"610adc82-acb1-4b98-877d-0bc6eb5e31db\") "
Sep 15 21:43:56.934703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache" (OuterVolumeSpecName: "model-cache") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.935149 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home" (OuterVolumeSpecName: "home") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.935149 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.934908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.935149 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.935314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935164    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:56.935314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935181    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:56.935314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935197    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:56.935314 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935208    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:56.935849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.935798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home" (OuterVolumeSpecName: "home") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.937080 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.937011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm" (OuterVolumeSpecName: "dshm") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.937250 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.937082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm" (OuterVolumeSpecName: "dshm") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:43:56.937466 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.937439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n" (OuterVolumeSpecName: "kube-api-access-zrw4n") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "kube-api-access-zrw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:56.937763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.937743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q" (OuterVolumeSpecName: "kube-api-access-lb29q") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "kube-api-access-lb29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:43:56.938894 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.938871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3514f173-0573-415b-8682-f6c971179d1e" (UID: "3514f173-0573-415b-8682-f6c971179d1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:56.939165 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:56.939135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "610adc82-acb1-4b98-877d-0bc6eb5e31db" (UID: "610adc82-acb1-4b98-877d-0bc6eb5e31db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036546    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036579    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036587    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610adc82-acb1-4b98-877d-0bc6eb5e31db-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036597    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrw4n\" (UniqueName: \"kubernetes.io/projected/3514f173-0573-415b-8682-f6c971179d1e-kube-api-access-zrw4n\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036609    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb29q\" (UniqueName: \"kubernetes.io/projected/610adc82-acb1-4b98-877d-0bc6eb5e31db-kube-api-access-lb29q\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036618    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610adc82-acb1-4b98-877d-0bc6eb5e31db-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036627    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3514f173-0573-415b-8682-f6c971179d1e-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.036637 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.036634    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3514f173-0573-415b-8682-f6c971179d1e-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:43:57.097735 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.097690    2578 generic.go:356] "Generic (PLEG): container finished" podID="3514f173-0573-415b-8682-f6c971179d1e" containerID="bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b" exitCode=0
Sep 15 21:43:57.097895 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.097774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"
Sep 15 21:43:57.097895 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.097777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" event={"ID":"3514f173-0573-415b-8682-f6c971179d1e","Type":"ContainerDied","Data":"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"}
Sep 15 21:43:57.097895 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.097826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm" event={"ID":"3514f173-0573-415b-8682-f6c971179d1e","Type":"ContainerDied","Data":"da72328d57110de72be131f7268bf12097849afcdbe93063b6b869fc1a53c367"}
Sep 15 21:43:57.097895 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.097850    2578 scope.go:117] "RemoveContainer" containerID="bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"
Sep 15 21:43:57.099326 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.099300    2578 generic.go:356] "Generic (PLEG): container finished" podID="610adc82-acb1-4b98-877d-0bc6eb5e31db" containerID="189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625" exitCode=0
Sep 15 21:43:57.099503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.099339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"
Sep 15 21:43:57.099503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.099384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" event={"ID":"610adc82-acb1-4b98-877d-0bc6eb5e31db","Type":"ContainerDied","Data":"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"}
Sep 15 21:43:57.099503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.099425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f" event={"ID":"610adc82-acb1-4b98-877d-0bc6eb5e31db","Type":"ContainerDied","Data":"e82b7a73e3321f2b54e29d431760ec7889bb07b4e19bd0500ae3c7662075e98a"}
Sep 15 21:43:57.106903 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.106886    2578 scope.go:117] "RemoveContainer" containerID="bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"
Sep 15 21:43:57.107224 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:57.107203    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b\": container with ID starting with bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b not found: ID does not exist" containerID="bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"
Sep 15 21:43:57.107332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.107231    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b"} err="failed to get container status \"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b\": rpc error: code = NotFound desc = could not find container \"bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b\": container with ID starting with bbc5b897d795b54a7d452be3efc3ec7008f20256ca69ede42762df5b2f215c3b not found: ID does not exist"
Sep 15 21:43:57.107332 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.107247    2578 scope.go:117] "RemoveContainer" containerID="189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"
Sep 15 21:43:57.115664 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.115645    2578 scope.go:117] "RemoveContainer" containerID="189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"
Sep 15 21:43:57.115977 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:43:57.115958    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625\": container with ID starting with 189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625 not found: ID does not exist" containerID="189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"
Sep 15 21:43:57.116037 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.115987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625"} err="failed to get container status \"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625\": rpc error: code = NotFound desc = could not find container \"189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625\": container with ID starting with 189de195b3392137924a19fb09f4d543579d8e9abd0c3f504dcc635a2d366625 not found: ID does not exist"
Sep 15 21:43:57.122173 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.122150    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:43:57.128785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.128755    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-w0-b99d7b-kserve-6d5cdcd478-ggg9f"]
Sep 15 21:43:57.141212 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.141183    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:43:57.145636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:57.145613    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b99d7b/distinct-2-w0-b99d7b-kserve-5cc875b69-jv5bm"]
Sep 15 21:43:58.075395 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:58.075354    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3514f173-0573-415b-8682-f6c971179d1e" path="/var/lib/kubelet/pods/3514f173-0573-415b-8682-f6c971179d1e/volumes"
Sep 15 21:43:58.075980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:43:58.075959    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610adc82-acb1-4b98-877d-0bc6eb5e31db" path="/var/lib/kubelet/pods/610adc82-acb1-4b98-877d-0bc6eb5e31db/volumes"
Sep 15 21:44:16.076435 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076402    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7754-6e268a"]
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076819    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356859e8-5908-4d4b-a2bc-8a9c204d7418" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076833    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="356859e8-5908-4d4b-a2bc-8a9c204d7418" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076844    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffc881e-e85e-4250-b7c7-69230edda6b8" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076851    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffc881e-e85e-4250-b7c7-69230edda6b8" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076858    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076864    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076871    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efb254f-4201-473e-ac54-62485172d790" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076876    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efb254f-4201-473e-ac54-62485172d790" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076885    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076890    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076898    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076905    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076915    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076923    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d6935b-0628-428a-909d-832e3d53d826" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076929    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d6935b-0628-428a-909d-832e3d53d826" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076942    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c230cea2-7fc4-47f8-aff2-5e21ff286046" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076947    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c230cea2-7fc4-47f8-aff2-5e21ff286046" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076957    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076962    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076968    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f391135-3881-4374-a4dc-ba1be26a6021" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076973    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f391135-3881-4374-a4dc-ba1be26a6021" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076980    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610adc82-acb1-4b98-877d-0bc6eb5e31db" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076985    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="610adc82-acb1-4b98-877d-0bc6eb5e31db" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076990    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3514f173-0573-415b-8682-f6c971179d1e" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.076995    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3514f173-0573-415b-8682-f6c971179d1e" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077056    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5bd7051-8a23-4306-a4cf-40d95ac44f50" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077064    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="610adc82-acb1-4b98-877d-0bc6eb5e31db" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077071    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffc881e-e85e-4250-b7c7-69230edda6b8" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077077    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f391135-3881-4374-a4dc-ba1be26a6021" containerName="payload-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077083    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077089    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3514f173-0573-415b-8682-f6c971179d1e" containerName="main"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077094    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9233d557-a6aa-4d6a-aecd-58cab5f517cd" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077099    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c230cea2-7fc4-47f8-aff2-5e21ff286046" containerName="payload-pre-processing"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077105    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efb254f-4201-473e-ac54-62485172d790" containerName="maas-api"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077110    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2dae7f9-0749-4b74-895d-d58402d943be" containerName="istio-proxy"
Sep 15 21:44:16.077114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e54cd9b-7a2e-43c4-b650-c5e7ae51a771" containerName="payload-processing"
Sep 15 21:44:16.079038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d6935b-0628-428a-909d-832e3d53d826" containerName="payload-pre-processing"
Sep 15 21:44:16.079038 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.077126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="356859e8-5908-4d4b-a2bc-8a9c204d7418" containerName="maas-api"
Sep 15 21:44:16.080072 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.080050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:16.080218 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.080194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-6e268a"]
Sep 15 21:44:16.081986 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.081962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\""
Sep 15 21:44:16.125662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.125627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdgm\" (UniqueName: \"kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm\") pod \"test-curl-7754-6e268a\" (UID: \"0ab807f4-c198-47bb-ba40-50efbb2f2ff7\") " pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:16.227256 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.227219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdgm\" (UniqueName: \"kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm\") pod \"test-curl-7754-6e268a\" (UID: \"0ab807f4-c198-47bb-ba40-50efbb2f2ff7\") " pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:16.235019 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.234988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdgm\" (UniqueName: \"kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm\") pod \"test-curl-7754-6e268a\" (UID: \"0ab807f4-c198-47bb-ba40-50efbb2f2ff7\") " pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:16.392979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.392890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:16.663343 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:16.663228    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-6e268a"]
Sep 15 21:44:16.665750 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:44:16.665706    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ab807f4_c198_47bb_ba40_50efbb2f2ff7.slice/crio-098ec824ad583608907c0715c85f3efa7b7de2a6dd1e5a949aac955165403989 WatchSource:0}: Error finding container 098ec824ad583608907c0715c85f3efa7b7de2a6dd1e5a949aac955165403989: Status 404 returned error can't find the container with id 098ec824ad583608907c0715c85f3efa7b7de2a6dd1e5a949aac955165403989
Sep 15 21:44:17.211370 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:17.211333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-6e268a" event={"ID":"0ab807f4-c198-47bb-ba40-50efbb2f2ff7","Type":"ContainerStarted","Data":"da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6"}
Sep 15 21:44:17.211370 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:17.211376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-6e268a" event={"ID":"0ab807f4-c198-47bb-ba40-50efbb2f2ff7","Type":"ContainerStarted","Data":"098ec824ad583608907c0715c85f3efa7b7de2a6dd1e5a949aac955165403989"}
Sep 15 21:44:17.227118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:17.227062    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7754-6e268a" podStartSLOduration=1.227047458 podStartE2EDuration="1.227047458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:17.224875219 +0000 UTC m=+1685.625848466" watchObservedRunningTime="2026-09-15 21:44:17.227047458 +0000 UTC m=+1685.628020757"
Sep 15 21:44:17.879992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:17.879864    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7754-6e268a"]
Sep 15 21:44:17.882572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:17.882539    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7754-6e268a"]
Sep 15 21:44:18.223218 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:44:18.223182    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7754-6e268a" container="test-curl-7754-6e268a"
Sep 15 21:44:19.228298 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:19.228256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7754-6e268a" podUID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" containerName="test-curl-7754-6e268a" containerID="cri-o://da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6" gracePeriod=2
Sep 15 21:44:21.418424 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:21.418397    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:21.479523 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:21.479485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctdgm\" (UniqueName: \"kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm\") pod \"0ab807f4-c198-47bb-ba40-50efbb2f2ff7\" (UID: \"0ab807f4-c198-47bb-ba40-50efbb2f2ff7\") "
Sep 15 21:44:21.481644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:21.481584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm" (OuterVolumeSpecName: "kube-api-access-ctdgm") pod "0ab807f4-c198-47bb-ba40-50efbb2f2ff7" (UID: "0ab807f4-c198-47bb-ba40-50efbb2f2ff7"). InnerVolumeSpecName "kube-api-access-ctdgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:21.581180 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:21.581140    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdgm\" (UniqueName: \"kubernetes.io/projected/0ab807f4-c198-47bb-ba40-50efbb2f2ff7-kube-api-access-ctdgm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:22.076134 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.076102    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" path="/var/lib/kubelet/pods/0ab807f4-c198-47bb-ba40-50efbb2f2ff7/volumes"
Sep 15 21:44:22.244589 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.244553    2578 generic.go:356] "Generic (PLEG): container finished" podID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" containerID="da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6" exitCode=137
Sep 15 21:44:22.244799 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.244593    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e268a"
Sep 15 21:44:22.244799 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.244658    2578 scope.go:117] "RemoveContainer" containerID="da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6"
Sep 15 21:44:22.253587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.253531    2578 scope.go:117] "RemoveContainer" containerID="da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6"
Sep 15 21:44:22.253926 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:44:22.253904    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6\": container with ID starting with da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6 not found: ID does not exist" containerID="da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6"
Sep 15 21:44:22.254014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:22.253936    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6"} err="failed to get container status \"da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6\": rpc error: code = NotFound desc = could not find container \"da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6\": container with ID starting with da8de55b6a2a8ce645ccd147c1f7ec942a7c2e3562b338dd64f1f6fd8e37c2f6 not found: ID does not exist"
Sep 15 21:44:28.399905 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:28.399858    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:44:36.125361 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.125265    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:44:36.125896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.125674    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" containerName="test-curl-7754-6e268a"
Sep 15 21:44:36.125896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.125685    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" containerName="test-curl-7754-6e268a"
Sep 15 21:44:36.125896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.125783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab807f4-c198-47bb-ba40-50efbb2f2ff7" containerName="test-curl-7754-6e268a"
Sep 15 21:44:36.130196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.130174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.132189 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.132166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"sub-test-model-59e243-kserve-self-signed-certs\""
Sep 15 21:44:36.138112 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.137960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:44:36.225939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.225897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.226117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.226005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.226117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.226062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.226117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.226096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.226236 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.226119    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.226236 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.226199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgsz\" (UniqueName: \"kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.326774 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.326954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.326954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.326954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.326954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgsz\" (UniqueName: \"kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.327159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.326971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.327241 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.327214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.327298 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.327250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.327298 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.327287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.329100 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.329078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.329270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.329250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.334912 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.334890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgsz\" (UniqueName: \"kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz\") pod \"sub-test-model-59e243-kserve-75f447698c-5xqnq\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") " pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.441329 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.441298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:36.711394 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:36.711369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:44:36.713667 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:44:36.713636    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod880c85c7_cfbf_47c2_9fc3_32c49a672e98.slice/crio-e190317423b62eb9192ee3d8d3f9e10b77f2188e36d98597f95c0ac95d5af02f WatchSource:0}: Error finding container e190317423b62eb9192ee3d8d3f9e10b77f2188e36d98597f95c0ac95d5af02f: Status 404 returned error can't find the container with id e190317423b62eb9192ee3d8d3f9e10b77f2188e36d98597f95c0ac95d5af02f
Sep 15 21:44:37.325731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:37.325694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" event={"ID":"880c85c7-cfbf-47c2-9fc3-32c49a672e98","Type":"ContainerStarted","Data":"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"}
Sep 15 21:44:37.326111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:37.325749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" event={"ID":"880c85c7-cfbf-47c2-9fc3-32c49a672e98","Type":"ContainerStarted","Data":"e190317423b62eb9192ee3d8d3f9e10b77f2188e36d98597f95c0ac95d5af02f"}
Sep 15 21:44:37.342942 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:37.342890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" podStartSLOduration=1.342873345 podStartE2EDuration="1.342873345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:37.341528052 +0000 UTC m=+1705.742501297" watchObservedRunningTime="2026-09-15 21:44:37.342873345 +0000 UTC m=+1705.743846597"
Sep 15 21:44:43.419845 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.419802    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerName="main" containerID="cri-o://795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad" gracePeriod=45
Sep 15 21:44:43.496790 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.496711    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused"
Sep 15 21:44:43.667403 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.667378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:44:43.799508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.799508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.799816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.799816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vmkc\" (UniqueName: \"kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.799816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799656    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.799816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home\") pod \"47e4ba9c-9173-4444-a442-ad170c76fdbe\" (UID: \"47e4ba9c-9173-4444-a442-ad170c76fdbe\") "
Sep 15 21:44:43.800054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:43.800054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.799967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:43.800054 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.800047    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:43.800211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.800065    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:43.800211 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.800077    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home" (OuterVolumeSpecName: "home") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:43.802023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.801995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:43.802132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.801995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm" (OuterVolumeSpecName: "dshm") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:43.802132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.802091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc" (OuterVolumeSpecName: "kube-api-access-8vmkc") pod "47e4ba9c-9173-4444-a442-ad170c76fdbe" (UID: "47e4ba9c-9173-4444-a442-ad170c76fdbe"). InnerVolumeSpecName "kube-api-access-8vmkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:43.901488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.901444    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:43.901488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.901483    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmkc\" (UniqueName: \"kubernetes.io/projected/47e4ba9c-9173-4444-a442-ad170c76fdbe-kube-api-access-8vmkc\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:43.901488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.901494    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47e4ba9c-9173-4444-a442-ad170c76fdbe-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:43.901788 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:43.901503    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47e4ba9c-9173-4444-a442-ad170c76fdbe-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:44:44.360970 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.360886    2578 generic.go:356] "Generic (PLEG): container finished" podID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerID="795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad" exitCode=0
Sep 15 21:44:44.360970 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.360926    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"
Sep 15 21:44:44.361159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.360966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" event={"ID":"47e4ba9c-9173-4444-a442-ad170c76fdbe","Type":"ContainerDied","Data":"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"}
Sep 15 21:44:44.361159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.361010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks" event={"ID":"47e4ba9c-9173-4444-a442-ad170c76fdbe","Type":"ContainerDied","Data":"3a38c8996847f3b70129a3ff194daf7e6bab5369da177de488fbf3a073c92cc5"}
Sep 15 21:44:44.361159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.361032    2578 scope.go:117] "RemoveContainer" containerID="795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"
Sep 15 21:44:44.372200 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.372175    2578 scope.go:117] "RemoveContainer" containerID="795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"
Sep 15 21:44:44.372506 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:44:44.372484    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad\": container with ID starting with 795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad not found: ID does not exist" containerID="795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"
Sep 15 21:44:44.372590 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.372515    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad"} err="failed to get container status \"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad\": rpc error: code = NotFound desc = could not find container \"795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad\": container with ID starting with 795825ac630c8c6ca35b308a454401de69fee05c847aa7ffa5ccacdb2346c8ad not found: ID does not exist"
Sep 15 21:44:44.376439 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.376412    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:44:44.379783 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:44.379760    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/auth-test-model-59e243-kserve-76b64b4db7-p42ks"]
Sep 15 21:44:46.076931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:46.076898    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" path="/var/lib/kubelet/pods/47e4ba9c-9173-4444-a442-ad170c76fdbe/volumes"
Sep 15 21:44:46.442215 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:46.442184    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:46.442420 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:46.442251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:46.454981 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:46.454953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:44:47.388684 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:44:47.388646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:45:00.158369 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.158331    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"]
Sep 15 21:45:00.158808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.158787    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerName="main"
Sep 15 21:45:00.158808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.158799    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerName="main"
Sep 15 21:45:00.158904 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.158897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e4ba9c-9173-4444-a442-ad170c76fdbe" containerName="main"
Sep 15 21:45:00.163800 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.163774    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"
Sep 15 21:45:00.169804 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.169779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sjzvg\""
Sep 15 21:45:00.175268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.175242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"
Sep 15 21:45:00.175430 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.175360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"]
Sep 15 21:45:00.449728 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.449691    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"]
Sep 15 21:45:00.452028 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:45:00.452002    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe6cebe_b991_4efc_83f1_9aad9c43ae92.slice/crio-6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b WatchSource:0}: Error finding container 6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b: Status 404 returned error can't find the container with id 6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b
Sep 15 21:45:00.453849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:00.453829    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:45:01.451791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:01.451737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j" event={"ID":"cbe6cebe-b991-4efc-83f1-9aad9c43ae92","Type":"ContainerStarted","Data":"6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b"}
Sep 15 21:45:03.464706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:03.464673    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbe6cebe-b991-4efc-83f1-9aad9c43ae92" containerID="0b238bf63bb8e74b1ee201cf8d3ea840099972c69d0ea9228866546485ff3d86" exitCode=0
Sep 15 21:45:03.465193 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:03.464755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j" event={"ID":"cbe6cebe-b991-4efc-83f1-9aad9c43ae92","Type":"ContainerDied","Data":"0b238bf63bb8e74b1ee201cf8d3ea840099972c69d0ea9228866546485ff3d86"}
Sep 15 21:45:04.606497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:04.606472    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"
Sep 15 21:45:05.477042 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:05.477010    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j"
Sep 15 21:45:05.477252 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:05.477050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-pfw2j" event={"ID":"cbe6cebe-b991-4efc-83f1-9aad9c43ae92","Type":"ContainerDied","Data":"6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b"}
Sep 15 21:45:05.477252 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:05.477100    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e54d84ebc4e4edd338f24f570d4c451246d837bcd6acfa8c720e97299c8ee0b"
Sep 15 21:45:41.751771 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.751734    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7754-6e3c8a"]
Sep 15 21:45:41.752247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.752154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbe6cebe-b991-4efc-83f1-9aad9c43ae92" containerName="cleanup"
Sep 15 21:45:41.752247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.752167    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbe6cebe-b991-4efc-83f1-9aad9c43ae92" containerName="cleanup"
Sep 15 21:45:41.752247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.752236    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbe6cebe-b991-4efc-83f1-9aad9c43ae92" containerName="cleanup"
Sep 15 21:45:41.760350 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.760320    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:41.761446 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.761422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-6e3c8a"]
Sep 15 21:45:41.762863 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.762838    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8rngv\""
Sep 15 21:45:41.882427 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.882385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gcx\" (UniqueName: \"kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx\") pod \"test-curl-7754-6e3c8a\" (UID: \"4a3f653c-4c78-4f22-a050-18e46379f67e\") " pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:41.983527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.983488    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gcx\" (UniqueName: \"kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx\") pod \"test-curl-7754-6e3c8a\" (UID: \"4a3f653c-4c78-4f22-a050-18e46379f67e\") " pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:41.992357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:41.992321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gcx\" (UniqueName: \"kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx\") pod \"test-curl-7754-6e3c8a\" (UID: \"4a3f653c-4c78-4f22-a050-18e46379f67e\") " pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:42.070912 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:42.070816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:42.341506 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:42.341484    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7754-6e3c8a"]
Sep 15 21:45:42.344160 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:45:42.344132    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a3f653c_4c78_4f22_a050_18e46379f67e.slice/crio-22d3477eb51bf41ad198e1b563c97300bddb1f0fb51ccd89a3f62ec430c4c2a5 WatchSource:0}: Error finding container 22d3477eb51bf41ad198e1b563c97300bddb1f0fb51ccd89a3f62ec430c4c2a5: Status 404 returned error can't find the container with id 22d3477eb51bf41ad198e1b563c97300bddb1f0fb51ccd89a3f62ec430c4c2a5
Sep 15 21:45:42.671890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:42.671798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-6e3c8a" event={"ID":"4a3f653c-4c78-4f22-a050-18e46379f67e","Type":"ContainerStarted","Data":"c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6"}
Sep 15 21:45:42.671890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:42.671840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7754-6e3c8a" event={"ID":"4a3f653c-4c78-4f22-a050-18e46379f67e","Type":"ContainerStarted","Data":"22d3477eb51bf41ad198e1b563c97300bddb1f0fb51ccd89a3f62ec430c4c2a5"}
Sep 15 21:45:42.685682 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:42.685628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7754-6e3c8a" podStartSLOduration=1.685610676 podStartE2EDuration="1.685610676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:42.684620325 +0000 UTC m=+1771.085593570" watchObservedRunningTime="2026-09-15 21:45:42.685610676 +0000 UTC m=+1771.086583922"
Sep 15 21:45:43.233532 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:43.233493    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7754-6e3c8a"]
Sep 15 21:45:43.237933 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:43.237901    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7754-6e3c8a"]
Sep 15 21:45:43.676876 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:45:43.676839    2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7754-6e3c8a" container="test-curl-7754-6e3c8a"
Sep 15 21:45:44.681634 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:44.681593    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7754-6e3c8a" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" containerName="test-curl-7754-6e3c8a" containerID="cri-o://c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6" gracePeriod=2
Sep 15 21:45:45.206867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:45.206830    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:45:46.857346 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:46.857321    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:46.936641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:46.936541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5gcx\" (UniqueName: \"kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx\") pod \"4a3f653c-4c78-4f22-a050-18e46379f67e\" (UID: \"4a3f653c-4c78-4f22-a050-18e46379f67e\") "
Sep 15 21:45:46.938690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:46.938651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx" (OuterVolumeSpecName: "kube-api-access-j5gcx") pod "4a3f653c-4c78-4f22-a050-18e46379f67e" (UID: "4a3f653c-4c78-4f22-a050-18e46379f67e"). InnerVolumeSpecName "kube-api-access-j5gcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:47.037628 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.037595    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gcx\" (UniqueName: \"kubernetes.io/projected/4a3f653c-4c78-4f22-a050-18e46379f67e-kube-api-access-j5gcx\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:45:47.701832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.701796    2578 generic.go:356] "Generic (PLEG): container finished" podID="4a3f653c-4c78-4f22-a050-18e46379f67e" containerID="c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6" exitCode=137
Sep 15 21:45:47.702102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.701835    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:47.702102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.701896    2578 scope.go:117] "RemoveContainer" containerID="c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6"
Sep 15 21:45:47.710952 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.710933    2578 scope.go:117] "RemoveContainer" containerID="c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6"
Sep 15 21:45:47.711249 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:45:47.711225    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6\": container with ID starting with c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6 not found: ID does not exist" containerID="c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6"
Sep 15 21:45:47.711320 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:47.711261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6"} err="failed to get container status \"c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6\": rpc error: code = NotFound desc = could not find container \"c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6\": container with ID starting with c8a5852ec1de33bce4ce4a795f7ec0b2fd6fc672563133bada1e91073da6b6d6 not found: ID does not exist"
Sep 15 21:45:47.711809 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:45:47.711785    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7754-6e3c8a\" is forbidden: User \"system:node:ip-10-0-135-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-105.ec2.internal' and this object" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" pod="opendatahub/test-curl-7754-6e3c8a"
Sep 15 21:45:48.076708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:48.076621    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" path="/var/lib/kubelet/pods/4a3f653c-4c78-4f22-a050-18e46379f67e/volumes"
Sep 15 21:45:49.632954 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.632920    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:45:49.633581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.633557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" containerName="test-curl-7754-6e3c8a"
Sep 15 21:45:49.633680 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.633583    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" containerName="test-curl-7754-6e3c8a"
Sep 15 21:45:49.633753 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.633689    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3f653c-4c78-4f22-a050-18e46379f67e" containerName="test-curl-7754-6e3c8a"
Sep 15 21:45:49.638257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.638231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.640557 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.640535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59e243\"/\"rate-test-model-59e243-kserve-self-signed-certs\""
Sep 15 21:45:49.654805 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.654771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:45:49.766689 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.766906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlwd\" (UniqueName: \"kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.766906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.766906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.766906 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.767073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.766906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868034 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.867994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlwd\" (UniqueName: \"kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868034 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868473 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868823 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.868823 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.868791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.870446 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.870423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.870566 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.870548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.875887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.875863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlwd\" (UniqueName: \"kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd\") pod \"rate-test-model-59e243-kserve-f75dc455-mw76z\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") " pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:49.947261 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:49.947220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:50.219318 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:50.219291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:45:50.221979 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:45:50.221951    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ddd1407_6429_4eec_b3d1_442922b404a6.slice/crio-ce5c9182f3498ab40f6681a1349a4017c6738b328f3a1fff1b4c8ae12e98f994 WatchSource:0}: Error finding container ce5c9182f3498ab40f6681a1349a4017c6738b328f3a1fff1b4c8ae12e98f994: Status 404 returned error can't find the container with id ce5c9182f3498ab40f6681a1349a4017c6738b328f3a1fff1b4c8ae12e98f994
Sep 15 21:45:50.719464 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:50.719429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" event={"ID":"8ddd1407-6429-4eec-b3d1-442922b404a6","Type":"ContainerStarted","Data":"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"}
Sep 15 21:45:50.719897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:50.719478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" event={"ID":"8ddd1407-6429-4eec-b3d1-442922b404a6","Type":"ContainerStarted","Data":"ce5c9182f3498ab40f6681a1349a4017c6738b328f3a1fff1b4c8ae12e98f994"}
Sep 15 21:45:50.740520 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:50.740461    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" podStartSLOduration=1.740440332 podStartE2EDuration="1.740440332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:50.739975654 +0000 UTC m=+1779.140948900" watchObservedRunningTime="2026-09-15 21:45:50.740440332 +0000 UTC m=+1779.141413584"
Sep 15 21:45:59.948428 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:59.948372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:59.948955 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:59.948445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:45:59.961390 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:45:59.961361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:46:00.226465 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.226348    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" podUID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" containerName="main" containerID="cri-o://b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f" gracePeriod=45
Sep 15 21:46:00.477003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.476936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:46:00.580796 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580752    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fgsz\" (UniqueName: \"kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.580796 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.581108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.581108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.581108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580885    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.581108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.580937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home\") pod \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\" (UID: \"880c85c7-cfbf-47c2-9fc3-32c49a672e98\") "
Sep 15 21:46:00.581311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.581311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache" (OuterVolumeSpecName: "model-cache") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.581311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581243    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home" (OuterVolumeSpecName: "home") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.581443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581370    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.581443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581390    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.581443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.581406    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.582943 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.582918    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz" (OuterVolumeSpecName: "kube-api-access-4fgsz") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "kube-api-access-4fgsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:00.583465 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.583438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:46:00.583465 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.583436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm" (OuterVolumeSpecName: "dshm") pod "880c85c7-cfbf-47c2-9fc3-32c49a672e98" (UID: "880c85c7-cfbf-47c2-9fc3-32c49a672e98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:46:00.682194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.682150    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/880c85c7-cfbf-47c2-9fc3-32c49a672e98-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.682194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.682186    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/880c85c7-cfbf-47c2-9fc3-32c49a672e98-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.682194 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.682197    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgsz\" (UniqueName: \"kubernetes.io/projected/880c85c7-cfbf-47c2-9fc3-32c49a672e98-kube-api-access-4fgsz\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:46:00.776225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.776126    2578 generic.go:356] "Generic (PLEG): container finished" podID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" containerID="b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f" exitCode=0
Sep 15 21:46:00.776225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.776170    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"
Sep 15 21:46:00.776225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.776176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" event={"ID":"880c85c7-cfbf-47c2-9fc3-32c49a672e98","Type":"ContainerDied","Data":"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"}
Sep 15 21:46:00.776225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.776225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq" event={"ID":"880c85c7-cfbf-47c2-9fc3-32c49a672e98","Type":"ContainerDied","Data":"e190317423b62eb9192ee3d8d3f9e10b77f2188e36d98597f95c0ac95d5af02f"}
Sep 15 21:46:00.776571 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.776249    2578 scope.go:117] "RemoveContainer" containerID="b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"
Sep 15 21:46:00.786867 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.786845    2578 scope.go:117] "RemoveContainer" containerID="b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"
Sep 15 21:46:00.787200 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:46:00.787181    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f\": container with ID starting with b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f not found: ID does not exist" containerID="b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"
Sep 15 21:46:00.787279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.787208    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f"} err="failed to get container status \"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f\": rpc error: code = NotFound desc = could not find container \"b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f\": container with ID starting with b389106e88ad6ed0f518f8b76007a11824d8d674ddf57f4d5e2435734b69000f not found: ID does not exist"
Sep 15 21:46:00.789084 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.789060    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:46:00.799081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.799055    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:46:00.804568 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:00.804543    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/sub-test-model-59e243-kserve-75f447698c-5xqnq"]
Sep 15 21:46:02.077212 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:02.077176    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" path="/var/lib/kubelet/pods/880c85c7-cfbf-47c2-9fc3-32c49a672e98/volumes"
Sep 15 21:46:19.933395 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.933361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:46:19.934023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.934003    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" containerName="main"
Sep 15 21:46:19.934076 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.934025    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" containerName="main"
Sep 15 21:46:19.934113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.934096    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="880c85c7-cfbf-47c2-9fc3-32c49a672e98" containerName="main"
Sep 15 21:46:19.937278 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.937260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:19.940979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.940956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"rate-test-model-38a3a3-kserve-self-signed-certs\""
Sep 15 21:46:19.940979 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.940975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"openshift-service-ca.crt\""
Sep 15 21:46:19.941157 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.940956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"kube-root-ca.crt\""
Sep 15 21:46:19.941157 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.940956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-38a3a3\"/\"default-dockercfg-xszmj\""
Sep 15 21:46:19.953254 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:19.953222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:46:20.071147 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.071147 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.071479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.071479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.071479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.071479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.071390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq96l\" (UniqueName: \"kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172433 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172653 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq96l\" (UniqueName: \"kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172653 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172653 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.172874 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.173000 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.172943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.173061 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.173048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.173199 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.173176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.174993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.174962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.175391 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.175373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.181090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.181070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq96l\" (UniqueName: \"kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l\") pod \"rate-test-model-38a3a3-kserve-658cf646cf-n7k7t\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") " pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.247391 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.247360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:20.517993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.517964    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:46:20.519764 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:46:20.519733    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06ebc3c5_83f6_4cfe_ba38_2b0e48b0328f.slice/crio-d99120c6e1fb1fb1cf27ab05ebe914f3fd65a60c8fd939ed1d6f4d053b9fec83 WatchSource:0}: Error finding container d99120c6e1fb1fb1cf27ab05ebe914f3fd65a60c8fd939ed1d6f4d053b9fec83: Status 404 returned error can't find the container with id d99120c6e1fb1fb1cf27ab05ebe914f3fd65a60c8fd939ed1d6f4d053b9fec83
Sep 15 21:46:20.884683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.884585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" event={"ID":"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f","Type":"ContainerStarted","Data":"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"}
Sep 15 21:46:20.884683 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.884632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" event={"ID":"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f","Type":"ContainerStarted","Data":"d99120c6e1fb1fb1cf27ab05ebe914f3fd65a60c8fd939ed1d6f4d053b9fec83"}
Sep 15 21:46:20.902326 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:20.902268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" podStartSLOduration=1.9022479890000001 podStartE2EDuration="1.902247989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:20.901318603 +0000 UTC m=+1809.302291850" watchObservedRunningTime="2026-09-15 21:46:20.902247989 +0000 UTC m=+1809.303221241"
Sep 15 21:46:30.248268 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:30.248226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:30.248777 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:30.248529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:30.260819 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:30.260793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:30.950594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:30.950562    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:46:54.648654 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:54.648620    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:46:55.708221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:46:55.708189    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:47:01.890817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.884416    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:47:01.890817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.890049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:01.898595 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.898575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b2c5fe\""
Sep 15 21:47:01.914238 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.914208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:47:01.914667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.914641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:01.914809 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.914682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlqx\" (UniqueName: \"kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:01.939251 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.939220    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:47:01.942707 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.942687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:01.944931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.944907    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b2c5fe-dockercfg-8tpgt\""
Sep 15 21:47:01.968859 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:01.968833    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:47:02.015878 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.015848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznf8\" (UniqueName: \"kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.016078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.015885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.016078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.015921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:02.016078 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.015988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlqx\" (UniqueName: \"kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:02.016585 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.016558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:02.033551 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.033518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlqx\" (UniqueName: \"kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx\") pod \"payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:02.117207 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.117167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznf8\" (UniqueName: \"kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.117207 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.117206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.117877 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.117857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.130454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.130429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznf8\" (UniqueName: \"kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8\") pod \"payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.201086 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.201052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:02.253830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.253447    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:02.476656 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.476628    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:47:02.478842 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:47:02.478815    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52682831_e206_4fc0_ae3f_b631fa6d12fb.slice/crio-f6c7ec133c10b6f173040c041d489359b1ab1499dc87036ef7eabc205d76cc34 WatchSource:0}: Error finding container f6c7ec133c10b6f173040c041d489359b1ab1499dc87036ef7eabc205d76cc34: Status 404 returned error can't find the container with id f6c7ec133c10b6f173040c041d489359b1ab1499dc87036ef7eabc205d76cc34
Sep 15 21:47:02.549759 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:02.549667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:47:02.552521 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:47:02.552495    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ebf170_bcfe_45db_a235_fa52d26b2b5c.slice/crio-b6fcfd484a79c2ffa8b7907a78c46ffa4064fd3da846b9a1fee2a1a500a93e88 WatchSource:0}: Error finding container b6fcfd484a79c2ffa8b7907a78c46ffa4064fd3da846b9a1fee2a1a500a93e88: Status 404 returned error can't find the container with id b6fcfd484a79c2ffa8b7907a78c46ffa4064fd3da846b9a1fee2a1a500a93e88
Sep 15 21:47:03.110275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.110232    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" event={"ID":"52682831-e206-4fc0-ae3f-b631fa6d12fb","Type":"ContainerStarted","Data":"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"}
Sep 15 21:47:03.110850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.110303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" event={"ID":"52682831-e206-4fc0-ae3f-b631fa6d12fb","Type":"ContainerStarted","Data":"f6c7ec133c10b6f173040c041d489359b1ab1499dc87036ef7eabc205d76cc34"}
Sep 15 21:47:03.110850 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.110442    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:03.111960 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.111936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" event={"ID":"16ebf170-bcfe-45db-a235-fa52d26b2b5c","Type":"ContainerStarted","Data":"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"}
Sep 15 21:47:03.112107 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.111976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" event={"ID":"16ebf170-bcfe-45db-a235-fa52d26b2b5c","Type":"ContainerStarted","Data":"b6fcfd484a79c2ffa8b7907a78c46ffa4064fd3da846b9a1fee2a1a500a93e88"}
Sep 15 21:47:03.112107 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.112008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:03.129087 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.129031    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" podStartSLOduration=2.12901539 podStartE2EDuration="2.12901539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:03.126000308 +0000 UTC m=+1851.526973555" watchObservedRunningTime="2026-09-15 21:47:03.12901539 +0000 UTC m=+1851.529988636"
Sep 15 21:47:03.140913 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:03.140853    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" podStartSLOduration=2.140833841 podStartE2EDuration="2.140833841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:03.139990195 +0000 UTC m=+1851.540963472" watchObservedRunningTime="2026-09-15 21:47:03.140833841 +0000 UTC m=+1851.541807090"
Sep 15 21:47:09.668644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.668575    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" podUID="8ddd1407-6429-4eec-b3d1-442922b404a6" containerName="main" containerID="cri-o://963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0" gracePeriod=45
Sep 15 21:47:09.925463 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.925401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:47:09.995751 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.995958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995761    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxlwd\" (UniqueName: \"kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.995958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.995958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.995958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.995958 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.995898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home\") pod \"8ddd1407-6429-4eec-b3d1-442922b404a6\" (UID: \"8ddd1407-6429-4eec-b3d1-442922b404a6\") "
Sep 15 21:47:09.996229 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996083    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:09.996229 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996175    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:09.996229 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home" (OuterVolumeSpecName: "home") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:09.996368 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996350    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:09.996406 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996374    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:09.996406 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.996389    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:09.997996 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.997962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:47:09.998110 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.998006    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm" (OuterVolumeSpecName: "dshm") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:09.998154 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:09.998114    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd" (OuterVolumeSpecName: "kube-api-access-xxlwd") pod "8ddd1407-6429-4eec-b3d1-442922b404a6" (UID: "8ddd1407-6429-4eec-b3d1-442922b404a6"). InnerVolumeSpecName "kube-api-access-xxlwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:10.097255 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.097215    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ddd1407-6429-4eec-b3d1-442922b404a6-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:10.097255 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.097254    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlwd\" (UniqueName: \"kubernetes.io/projected/8ddd1407-6429-4eec-b3d1-442922b404a6-kube-api-access-xxlwd\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:10.097255 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.097264    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ddd1407-6429-4eec-b3d1-442922b404a6-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:10.150869 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.150830    2578 generic.go:356] "Generic (PLEG): container finished" podID="8ddd1407-6429-4eec-b3d1-442922b404a6" containerID="963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0" exitCode=0
Sep 15 21:47:10.151008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.150873    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"
Sep 15 21:47:10.151008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.150915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" event={"ID":"8ddd1407-6429-4eec-b3d1-442922b404a6","Type":"ContainerDied","Data":"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"}
Sep 15 21:47:10.151008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.150962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z" event={"ID":"8ddd1407-6429-4eec-b3d1-442922b404a6","Type":"ContainerDied","Data":"ce5c9182f3498ab40f6681a1349a4017c6738b328f3a1fff1b4c8ae12e98f994"}
Sep 15 21:47:10.151008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.150980    2578 scope.go:117] "RemoveContainer" containerID="963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"
Sep 15 21:47:10.159854 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.159835    2578 scope.go:117] "RemoveContainer" containerID="963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"
Sep 15 21:47:10.160106 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:10.160089    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0\": container with ID starting with 963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0 not found: ID does not exist" containerID="963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"
Sep 15 21:47:10.160168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.160112    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0"} err="failed to get container status \"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0\": rpc error: code = NotFound desc = could not find container \"963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0\": container with ID starting with 963f18ecdef643a7f4f04ca3ee56c10f96dc7629708329f89cbfc9ed67dcd5f0 not found: ID does not exist"
Sep 15 21:47:10.165988 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.165959    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:47:10.169667 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.169646    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59e243/rate-test-model-59e243-kserve-f75dc455-mw76z"]
Sep 15 21:47:10.729606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.729562    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerName="main" containerID="cri-o://41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199" gracePeriod=45
Sep 15 21:47:10.980272 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:10.980199    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:47:11.007097 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007258 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq96l\" (UniqueName: \"kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007258 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007258 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007326    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir\") pod \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\" (UID: \"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f\") "
Sep 15 21:47:11.007431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home" (OuterVolumeSpecName: "home") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:11.007587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache" (OuterVolumeSpecName: "model-cache") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:11.007676 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:11.007823 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007798    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.007823 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007822    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.008022 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.007836    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.009530 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.009508    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm" (OuterVolumeSpecName: "dshm") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:11.009750 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.009711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l" (OuterVolumeSpecName: "kube-api-access-fq96l") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "kube-api-access-fq96l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:11.009857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.009752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" (UID: "06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:47:11.108688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.108648    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.108688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.108682    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.108688 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.108693    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq96l\" (UniqueName: \"kubernetes.io/projected/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f-kube-api-access-fq96l\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:47:11.157644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.157609    2578 generic.go:356] "Generic (PLEG): container finished" podID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerID="41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199" exitCode=0
Sep 15 21:47:11.157818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.157655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"
Sep 15 21:47:11.157818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.157694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" event={"ID":"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f","Type":"ContainerDied","Data":"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"}
Sep 15 21:47:11.157818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.157769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" event={"ID":"06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f","Type":"ContainerDied","Data":"d99120c6e1fb1fb1cf27ab05ebe914f3fd65a60c8fd939ed1d6f4d053b9fec83"}
Sep 15 21:47:11.157818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.157790    2578 scope.go:117] "RemoveContainer" containerID="41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"
Sep 15 21:47:11.169200 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.169176    2578 scope.go:117] "RemoveContainer" containerID="41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"
Sep 15 21:47:11.169487 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:11.169467    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199\": container with ID starting with 41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199 not found: ID does not exist" containerID="41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"
Sep 15 21:47:11.169560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.169494    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199"} err="failed to get container status \"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199\": rpc error: code = NotFound desc = could not find container \"41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199\": container with ID starting with 41861fe8d6c28421b6c8f453500e2598229166f91805420c10bf9a7bf1def199 not found: ID does not exist"
Sep 15 21:47:11.181299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.181275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:47:11.184584 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:11.184561    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t"]
Sep 15 21:47:12.081033 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:12.080995    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" path="/var/lib/kubelet/pods/06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f/volumes"
Sep 15 21:47:12.081587 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:12.081569    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddd1407-6429-4eec-b3d1-442922b404a6" path="/var/lib/kubelet/pods/8ddd1407-6429-4eec-b3d1-442922b404a6/volumes"
Sep 15 21:47:14.119084 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:14.119047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:47:14.119475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:14.119167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:47:15.940807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:15.940746    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-38a3a3/rate-test-model-38a3a3-kserve-658cf646cf-n7k7t" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:47:30.493861 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.493824    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:47:30.494335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddd1407-6429-4eec-b3d1-442922b404a6" containerName="main"
Sep 15 21:47:30.494335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494292    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddd1407-6429-4eec-b3d1-442922b404a6" containerName="main"
Sep 15 21:47:30.494335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494307    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerName="main"
Sep 15 21:47:30.494335 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494313    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerName="main"
Sep 15 21:47:30.494477 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494393    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddd1407-6429-4eec-b3d1-442922b404a6" containerName="main"
Sep 15 21:47:30.494477 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.494405    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ebc3c5-83f6-4cfe-ba38-2b0e48b0328f" containerName="main"
Sep 15 21:47:30.497755 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.497736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.499851 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.499829    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:47:30.499983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.499863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b0ba80-openshift-default-dockercfg-f5d47\""
Sep 15 21:47:30.515225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.515196    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:47:30.604117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjcsr\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604331 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604463 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604463 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.604529 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.604466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.705808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.705768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.705808 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.705811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.705837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706279 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjcsr\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706443 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706622 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.706690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.706654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.707046 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.707011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.708321 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.708297    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.708596 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.708578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.713441 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.713415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.713857 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.713839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjcsr\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr\") pod \"e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") " pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:30.808741 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:30.808628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:31.087991 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.087962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:47:31.089865 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:47:31.089842    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6789f62_8901_4877_8a01_22b4b991188d.slice/crio-3f56332536d3303494c9f3b2cef24e4d0b032f71c767f9210b2e477d35331ad4 WatchSource:0}: Error finding container 3f56332536d3303494c9f3b2cef24e4d0b032f71c767f9210b2e477d35331ad4: Status 404 returned error can't find the container with id 3f56332536d3303494c9f3b2cef24e4d0b032f71c767f9210b2e477d35331ad4
Sep 15 21:47:31.092431 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.092395    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:31.092520 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.092467    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:31.092520 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.092498    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:31.265756 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.265708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerStarted","Data":"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"}
Sep 15 21:47:31.265972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.265784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerStarted","Data":"3f56332536d3303494c9f3b2cef24e4d0b032f71c767f9210b2e477d35331ad4"}
Sep 15 21:47:31.289644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.289580    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podStartSLOduration=1.289563577 podStartE2EDuration="1.289563577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:31.287230308 +0000 UTC m=+1879.688203557" watchObservedRunningTime="2026-09-15 21:47:31.289563577 +0000 UTC m=+1879.690536823"
Sep 15 21:47:31.808884 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:31.808844    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:32.814373 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:32.814341    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:33.278250 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:33.278221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:33.279119 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:33.279096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:35.942517 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.942438    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:47:35.947486 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.947455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:35.949322 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.949297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b0ba80-serving-cert\""
Sep 15 21:47:35.956765 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.956733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:47:35.958652 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.958628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:35.958779 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.958662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:35.958836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.958793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:35.958881 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:35.958836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2d4\" (UniqueName: \"kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.059480 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.059449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.059480 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.059486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.059708 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.059549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.060006 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.059573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2d4\" (UniqueName: \"kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.060258 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:36.060213    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-serving-cert: secret "maas-api-e2e-ipp-b-b0ba80-serving-cert" not found
Sep 15 21:47:36.060405 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:36.060316    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls podName:5b5aa8b5-5d69-46a7-952e-2eb1e00f621a nodeName:}" failed. No retries permitted until 2026-09-15 21:47:36.560293274 +0000 UTC m=+1884.961266500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls") pod "maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a") : secret "maas-api-e2e-ipp-b-b0ba80-serving-cert" not found
Sep 15 21:47:36.060500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.060483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.063259 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.063235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.069731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.069696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2d4\" (UniqueName: \"kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.565254 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:36.565214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:36.565437 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:36.565359    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-serving-cert: secret "maas-api-e2e-ipp-b-b0ba80-serving-cert" not found
Sep 15 21:47:36.565437 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:47:36.565428    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls podName:5b5aa8b5-5d69-46a7-952e-2eb1e00f621a nodeName:}" failed. No retries permitted until 2026-09-15 21:47:37.56541183 +0000 UTC m=+1885.966385054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls") pod "maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a") : secret "maas-api-e2e-ipp-b-b0ba80-serving-cert" not found
Sep 15 21:47:37.576607 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:37.576569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:37.579017 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:37.578990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:37.761247 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:37.761208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:38.038407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:38.038375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:47:38.040619 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:47:38.040588    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5aa8b5_5d69_46a7_952e_2eb1e00f621a.slice/crio-7ecd6f8dc46c22dad7747ec380d6c260f40168afe53689b2c6f03c647103b5ee WatchSource:0}: Error finding container 7ecd6f8dc46c22dad7747ec380d6c260f40168afe53689b2c6f03c647103b5ee: Status 404 returned error can't find the container with id 7ecd6f8dc46c22dad7747ec380d6c260f40168afe53689b2c6f03c647103b5ee
Sep 15 21:47:38.309870 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:38.309784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" event={"ID":"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a","Type":"ContainerStarted","Data":"7ecd6f8dc46c22dad7747ec380d6c260f40168afe53689b2c6f03c647103b5ee"}
Sep 15 21:47:39.317759 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:39.317698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" event={"ID":"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a","Type":"ContainerStarted","Data":"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"}
Sep 15 21:47:39.320250 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:39.317846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:39.336985 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:39.336929    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" podStartSLOduration=3.9955358949999997 podStartE2EDuration="4.336912183s" totalImagesPullingTime="341.376288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:39.333687341 +0000 UTC m=+1887.734660588" watchObservedRunningTime="2026-09-15 21:47:39.336912183 +0000 UTC m=+1887.737885429"
Sep 15 21:47:45.329405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:45.329374    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:47:49.319911 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:49.319867    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 21:47:49.320304 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:49.319938    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 21:47:51.384580 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.384541    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6789f62-8901-4877-8a01-22b4b991188d" containerID="f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d" exitCode=0
Sep 15 21:47:51.385041 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.384615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerDied","Data":"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"}
Sep 15 21:47:51.385180 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.385160    2578 scope.go:117] "RemoveContainer" containerID="f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"
Sep 15 21:47:51.386366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.386338    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:51.386439 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.386406    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:51.386479 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:51.386436    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 15 21:47:52.394731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:52.394688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerStarted","Data":"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"}
Sep 15 21:47:52.809399 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:52.809353    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:52.809804 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:52.809776    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 15 21:47:52.809932 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:52.809835    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 15 21:47:53.809811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:53.809769    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 15 21:47:53.810200 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:53.809847    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 15 21:47:54.809887 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:54.809848    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 15 21:47:54.810308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:54.809935    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 15 21:47:55.809941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:55.809902    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 15 21:47:55.810342 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:55.809970    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 15 21:47:56.809983 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:56.809942    2578 patch_prober.go:28] interesting pod/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Sep 15 21:47:56.810414 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:56.810027    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Sep 15 21:47:57.813447 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:57.813413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:57.813945 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:57.813696    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:47:57.814499 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:47:57.814481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:48:54.308469 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:48:54.308437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh_16ebf170-bcfe-45db-a235-fa52d26b2b5c/payload-processing/0.log"
Sep 15 21:49:01.904837 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.904794    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:01.907524 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.907506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:01.911005 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.910985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5dc07b-serving-cert\""
Sep 15 21:49:01.948617 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.948577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:01.948811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.948637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:01.948811 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.948690    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnl5\" (UniqueName: \"kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:01.948909 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.948816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:01.962911 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:01.962876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:02.049644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.049609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.049644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.049645    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnl5\" (UniqueName: \"kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.049873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.049709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.049873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.049780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.049946 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:49:02.049886    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-serving-cert: secret "maas-api-e2e-ipp-cleanup-5dc07b-serving-cert" not found
Sep 15 21:49:02.049988 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:49:02.049952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls podName:8ae06ab8-b13a-46d1-934c-133febe03e54 nodeName:}" failed. No retries permitted until 2026-09-15 21:49:02.549932761 +0000 UTC m=+1970.950905985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" (UID: "8ae06ab8-b13a-46d1-934c-133febe03e54") : secret "maas-api-e2e-ipp-cleanup-5dc07b-serving-cert" not found
Sep 15 21:49:02.050480 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.050453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.052159 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.052128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.060821 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.060798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnl5\" (UniqueName: \"kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.554531 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.554493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.556982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.556954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.818450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.818359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:02.841757 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.841699    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:02.845658 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.845632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:02.847769 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.847735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5dc07b-dockercfg-bsxnt\""
Sep 15 21:49:02.847769 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.847745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5dc07b\""
Sep 15 21:49:02.854841 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.854800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:02.959065 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.959028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46x7\" (UniqueName: \"kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:02.959456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:02.959106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.059817 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.059779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46x7\" (UniqueName: \"kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.060009 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.059862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.060541 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.060520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.068307 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.068280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46x7\" (UniqueName: \"kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7\") pod \"payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.102461 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.102433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:03.105187 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:49:03.105161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae06ab8_b13a_46d1_934c_133febe03e54.slice/crio-24ca7d348c0b4371007621fdd661bdcd5699813267e75202caf5990aa2b127f1 WatchSource:0}: Error finding container 24ca7d348c0b4371007621fdd661bdcd5699813267e75202caf5990aa2b127f1: Status 404 returned error can't find the container with id 24ca7d348c0b4371007621fdd661bdcd5699813267e75202caf5990aa2b127f1
Sep 15 21:49:03.164495 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.164464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.462505 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.462471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:03.466198 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:49:03.466168    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131dcf3e_fe7f_4f0a_bff9_9592d73f4002.slice/crio-5b7815f059d9a9d0ba684bff6c9ade6e6d1a4eecde2ccbf58e9b619afc606aca WatchSource:0}: Error finding container 5b7815f059d9a9d0ba684bff6c9ade6e6d1a4eecde2ccbf58e9b619afc606aca: Status 404 returned error can't find the container with id 5b7815f059d9a9d0ba684bff6c9ade6e6d1a4eecde2ccbf58e9b619afc606aca
Sep 15 21:49:03.774444 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.774346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" event={"ID":"8ae06ab8-b13a-46d1-934c-133febe03e54","Type":"ContainerStarted","Data":"24ca7d348c0b4371007621fdd661bdcd5699813267e75202caf5990aa2b127f1"}
Sep 15 21:49:03.776117 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.776084    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" event={"ID":"131dcf3e-fe7f-4f0a-bff9-9592d73f4002","Type":"ContainerStarted","Data":"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"}
Sep 15 21:49:03.776275 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.776143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" event={"ID":"131dcf3e-fe7f-4f0a-bff9-9592d73f4002","Type":"ContainerStarted","Data":"5b7815f059d9a9d0ba684bff6c9ade6e6d1a4eecde2ccbf58e9b619afc606aca"}
Sep 15 21:49:03.776347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.776284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:03.795606 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:03.795546    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" podStartSLOduration=1.7955316049999999 podStartE2EDuration="1.795531605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:49:03.793414783 +0000 UTC m=+1972.194388042" watchObservedRunningTime="2026-09-15 21:49:03.795531605 +0000 UTC m=+1972.196504852"
Sep 15 21:49:04.783318 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:04.783280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" event={"ID":"8ae06ab8-b13a-46d1-934c-133febe03e54","Type":"ContainerStarted","Data":"c5d996aaebb025b659dc178db527deb4a29ce50846d6ee6542a5a8a767b8037c"}
Sep 15 21:49:04.783789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:04.783596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:04.803116 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:04.803054    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" podStartSLOduration=3.025389895 podStartE2EDuration="3.803036238s" totalImagesPullingTime="777.646343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:49:04.800886795 +0000 UTC m=+1973.201860035" watchObservedRunningTime="2026-09-15 21:49:04.803036238 +0000 UTC m=+1973.204009484"
Sep 15 21:49:10.794957 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:10.794926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:14.785849 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:14.785799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:39.851826 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.851777    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:39.852423 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.852080    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" podUID="8ae06ab8-b13a-46d1-934c-133febe03e54" containerName="maas-api" containerID="cri-o://c5d996aaebb025b659dc178db527deb4a29ce50846d6ee6542a5a8a767b8037c" gracePeriod=30
Sep 15 21:49:39.909639 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.909603    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:39.909897 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.909868    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" podUID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" containerName="payload-processing" containerID="cri-o://1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133" gracePeriod=30
Sep 15 21:49:39.983816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.983710    2578 generic.go:356] "Generic (PLEG): container finished" podID="8ae06ab8-b13a-46d1-934c-133febe03e54" containerID="c5d996aaebb025b659dc178db527deb4a29ce50846d6ee6542a5a8a767b8037c" exitCode=0
Sep 15 21:49:39.984008 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:39.983893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" event={"ID":"8ae06ab8-b13a-46d1-934c-133febe03e54","Type":"ContainerDied","Data":"c5d996aaebb025b659dc178db527deb4a29ce50846d6ee6542a5a8a767b8037c"}
Sep 15 21:49:40.161427 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.161399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:40.202090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.202064    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:40.249771 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.249707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs\") pod \"8ae06ab8-b13a-46d1-934c-133febe03e54\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") "
Sep 15 21:49:40.249974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.249785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") pod \"8ae06ab8-b13a-46d1-934c-133febe03e54\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") "
Sep 15 21:49:40.249974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.249832    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmnl5\" (UniqueName: \"kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5\") pod \"8ae06ab8-b13a-46d1-934c-133febe03e54\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") "
Sep 15 21:49:40.249974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.249888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca\") pod \"8ae06ab8-b13a-46d1-934c-133febe03e54\" (UID: \"8ae06ab8-b13a-46d1-934c-133febe03e54\") "
Sep 15 21:49:40.250572 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.250498    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ae06ab8-b13a-46d1-934c-133febe03e54" (UID: "8ae06ab8-b13a-46d1-934c-133febe03e54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:49:40.252679 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.252642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ae06ab8-b13a-46d1-934c-133febe03e54" (UID: "8ae06ab8-b13a-46d1-934c-133febe03e54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:49:40.252830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.252737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5" (OuterVolumeSpecName: "kube-api-access-vmnl5") pod "8ae06ab8-b13a-46d1-934c-133febe03e54" (UID: "8ae06ab8-b13a-46d1-934c-133febe03e54"). InnerVolumeSpecName "kube-api-access-vmnl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:40.252830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.252794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ae06ab8-b13a-46d1-934c-133febe03e54" (UID: "8ae06ab8-b13a-46d1-934c-133febe03e54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:49:40.350931 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.350902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x46x7\" (UniqueName: \"kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7\") pod \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") "
Sep 15 21:49:40.351118 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume\") pod \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\" (UID: \"131dcf3e-fe7f-4f0a-bff9-9592d73f4002\") "
Sep 15 21:49:40.351311 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351293    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae06ab8-b13a-46d1-934c-133febe03e54-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.351377 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351315    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.351377 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351329    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ae06ab8-b13a-46d1-934c-133febe03e54-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.351377 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351341    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnl5\" (UniqueName: \"kubernetes.io/projected/8ae06ab8-b13a-46d1-934c-133febe03e54-kube-api-access-vmnl5\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.351487 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.351370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "131dcf3e-fe7f-4f0a-bff9-9592d73f4002" (UID: "131dcf3e-fe7f-4f0a-bff9-9592d73f4002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:49:40.352842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.352821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7" (OuterVolumeSpecName: "kube-api-access-x46x7") pod "131dcf3e-fe7f-4f0a-bff9-9592d73f4002" (UID: "131dcf3e-fe7f-4f0a-bff9-9592d73f4002"). InnerVolumeSpecName "kube-api-access-x46x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:40.452761 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.452697    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46x7\" (UniqueName: \"kubernetes.io/projected/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-kube-api-access-x46x7\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.452761 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.452757    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/131dcf3e-fe7f-4f0a-bff9-9592d73f4002-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.991489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.991461    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"
Sep 15 21:49:40.991489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.991474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx" event={"ID":"8ae06ab8-b13a-46d1-934c-133febe03e54","Type":"ContainerDied","Data":"24ca7d348c0b4371007621fdd661bdcd5699813267e75202caf5990aa2b127f1"}
Sep 15 21:49:40.992044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.991543    2578 scope.go:117] "RemoveContainer" containerID="c5d996aaebb025b659dc178db527deb4a29ce50846d6ee6542a5a8a767b8037c"
Sep 15 21:49:40.993023 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.992998    2578 generic.go:356] "Generic (PLEG): container finished" podID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" containerID="1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133" exitCode=0
Sep 15 21:49:40.993093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.993031    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"
Sep 15 21:49:40.993093 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.993071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" event={"ID":"131dcf3e-fe7f-4f0a-bff9-9592d73f4002","Type":"ContainerDied","Data":"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"}
Sep 15 21:49:40.993188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:40.993119    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv" event={"ID":"131dcf3e-fe7f-4f0a-bff9-9592d73f4002","Type":"ContainerDied","Data":"5b7815f059d9a9d0ba684bff6c9ade6e6d1a4eecde2ccbf58e9b619afc606aca"}
Sep 15 21:49:41.001854 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.001830    2578 scope.go:117] "RemoveContainer" containerID="1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"
Sep 15 21:49:41.010245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.010226    2578 scope.go:117] "RemoveContainer" containerID="1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"
Sep 15 21:49:41.010523 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:49:41.010500    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133\": container with ID starting with 1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133 not found: ID does not exist" containerID="1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"
Sep 15 21:49:41.010576 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.010533    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133"} err="failed to get container status \"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133\": rpc error: code = NotFound desc = could not find container \"1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133\": container with ID starting with 1113ac2ee7273b1f7a6a6f1e3c5b74ce870fa37fbd75cb5d65855fabf1e97133 not found: ID does not exist"
Sep 15 21:49:41.015782 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.015750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:41.021446 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.021398    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5dc07b-676d5fd4f5-7xhhx"]
Sep 15 21:49:41.029915 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.029886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:41.033389 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:41.033363    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5dc07b-66f4d958c7-hl8jv"]
Sep 15 21:49:42.076972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:42.076942    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" path="/var/lib/kubelet/pods/131dcf3e-fe7f-4f0a-bff9-9592d73f4002/volumes"
Sep 15 21:49:42.077355 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:49:42.077334    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae06ab8-b13a-46d1-934c-133febe03e54" path="/var/lib/kubelet/pods/8ae06ab8-b13a-46d1-934c-133febe03e54/volumes"
Sep 15 21:50:06.192100 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.192058    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:50:06.192603 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.192332    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" podUID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" containerName="payload-processing" containerID="cri-o://0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e" gracePeriod=30
Sep 15 21:50:06.199700 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.199667    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:50:06.199947 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.199895    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" podUID="52682831-e206-4fc0-ae3f-b631fa6d12fb" containerName="payload-pre-processing" containerID="cri-o://fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46" gracePeriod=30
Sep 15 21:50:06.455365 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.455296    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:50:06.488508 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.488483    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:50:06.518882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.518846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vznf8\" (UniqueName: \"kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8\") pod \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") "
Sep 15 21:50:06.519089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.518908    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume\") pod \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\" (UID: \"16ebf170-bcfe-45db-a235-fa52d26b2b5c\") "
Sep 15 21:50:06.519301 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.519277    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16ebf170-bcfe-45db-a235-fa52d26b2b5c" (UID: "16ebf170-bcfe-45db-a235-fa52d26b2b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:50:06.520972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.520950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8" (OuterVolumeSpecName: "kube-api-access-vznf8") pod "16ebf170-bcfe-45db-a235-fa52d26b2b5c" (UID: "16ebf170-bcfe-45db-a235-fa52d26b2b5c"). InnerVolumeSpecName "kube-api-access-vznf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:50:06.620641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.620591    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume\") pod \"52682831-e206-4fc0-ae3f-b631fa6d12fb\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") "
Sep 15 21:50:06.620876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.620660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxlqx\" (UniqueName: \"kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx\") pod \"52682831-e206-4fc0-ae3f-b631fa6d12fb\" (UID: \"52682831-e206-4fc0-ae3f-b631fa6d12fb\") "
Sep 15 21:50:06.621125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.621099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52682831-e206-4fc0-ae3f-b631fa6d12fb" (UID: "52682831-e206-4fc0-ae3f-b631fa6d12fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:50:06.621167 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.621118    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznf8\" (UniqueName: \"kubernetes.io/projected/16ebf170-bcfe-45db-a235-fa52d26b2b5c-kube-api-access-vznf8\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:50:06.621167 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.621138    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ebf170-bcfe-45db-a235-fa52d26b2b5c-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:50:06.622778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.622752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx" (OuterVolumeSpecName: "kube-api-access-sxlqx") pod "52682831-e206-4fc0-ae3f-b631fa6d12fb" (UID: "52682831-e206-4fc0-ae3f-b631fa6d12fb"). InnerVolumeSpecName "kube-api-access-sxlqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:50:06.722499 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.722402    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52682831-e206-4fc0-ae3f-b631fa6d12fb-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:50:06.722499 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:06.722441    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlqx\" (UniqueName: \"kubernetes.io/projected/52682831-e206-4fc0-ae3f-b631fa6d12fb-kube-api-access-sxlqx\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:50:07.134225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.134134    2578 generic.go:356] "Generic (PLEG): container finished" podID="52682831-e206-4fc0-ae3f-b631fa6d12fb" containerID="fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46" exitCode=0
Sep 15 21:50:07.134225 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.134180    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"
Sep 15 21:50:07.134475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.134224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" event={"ID":"52682831-e206-4fc0-ae3f-b631fa6d12fb","Type":"ContainerDied","Data":"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"}
Sep 15 21:50:07.134475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.134282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55" event={"ID":"52682831-e206-4fc0-ae3f-b631fa6d12fb","Type":"ContainerDied","Data":"f6c7ec133c10b6f173040c041d489359b1ab1499dc87036ef7eabc205d76cc34"}
Sep 15 21:50:07.134475 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.134314    2578 scope.go:117] "RemoveContainer" containerID="fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"
Sep 15 21:50:07.135627 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.135607    2578 generic.go:356] "Generic (PLEG): container finished" podID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" containerID="0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e" exitCode=0
Sep 15 21:50:07.135737 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.135649    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"
Sep 15 21:50:07.135791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.135649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" event={"ID":"16ebf170-bcfe-45db-a235-fa52d26b2b5c","Type":"ContainerDied","Data":"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"}
Sep 15 21:50:07.135791 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.135773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh" event={"ID":"16ebf170-bcfe-45db-a235-fa52d26b2b5c","Type":"ContainerDied","Data":"b6fcfd484a79c2ffa8b7907a78c46ffa4064fd3da846b9a1fee2a1a500a93e88"}
Sep 15 21:50:07.148281 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.148264    2578 scope.go:117] "RemoveContainer" containerID="fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"
Sep 15 21:50:07.148585 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:50:07.148567    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46\": container with ID starting with fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46 not found: ID does not exist" containerID="fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"
Sep 15 21:50:07.148662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.148591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46"} err="failed to get container status \"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46\": rpc error: code = NotFound desc = could not find container \"fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46\": container with ID starting with fe6c9a22ff05455c4a66595e96c5377cae57772b9cb047071fd92d41b3230e46 not found: ID does not exist"
Sep 15 21:50:07.148662 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.148607    2578 scope.go:117] "RemoveContainer" containerID="0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"
Sep 15 21:50:07.157230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.157206    2578 scope.go:117] "RemoveContainer" containerID="0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"
Sep 15 21:50:07.157513 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:50:07.157493    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e\": container with ID starting with 0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e not found: ID does not exist" containerID="0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"
Sep 15 21:50:07.157581 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.157524    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e"} err="failed to get container status \"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e\": rpc error: code = NotFound desc = could not find container \"0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e\": container with ID starting with 0e6751e2280fba92a3fb3172c129c600f2e30e748d95f2f28cd16ca1bf52161e not found: ID does not exist"
Sep 15 21:50:07.159501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.159477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:50:07.161594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.161574    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b2c5fe-6f6f6c7854-9ch55"]
Sep 15 21:50:07.173769 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.173746    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:50:07.182818 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:07.182796    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b2c5fe-6855d78fc5-d9vvh"]
Sep 15 21:50:08.076108 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:08.076073    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" path="/var/lib/kubelet/pods/16ebf170-bcfe-45db-a235-fa52d26b2b5c/volumes"
Sep 15 21:50:08.076500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:50:08.076450    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52682831-e206-4fc0-ae3f-b631fa6d12fb" path="/var/lib/kubelet/pods/52682831-e206-4fc0-ae3f-b631fa6d12fb/volumes"
Sep 15 21:51:06.491509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.491467    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:51:06.492125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.491677    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" podUID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" containerName="maas-api" containerID="cri-o://49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d" gracePeriod=30
Sep 15 21:51:06.749146 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.749076    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:51:06.823832 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.823799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs\") pod \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") "
Sep 15 21:51:06.824014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.823870    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") pod \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") "
Sep 15 21:51:06.824014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.823910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz2d4\" (UniqueName: \"kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4\") pod \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") "
Sep 15 21:51:06.824014 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.823941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca\") pod \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\" (UID: \"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a\") "
Sep 15 21:51:06.824347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.824314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:06.825999 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.825965    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:06.826112 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.826041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4" (OuterVolumeSpecName: "kube-api-access-sz2d4") pod "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a"). InnerVolumeSpecName "kube-api-access-sz2d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:06.826112 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.826100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" (UID: "5b5aa8b5-5d69-46a7-952e-2eb1e00f621a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:06.925220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.925179    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:06.925220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.925213    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2d4\" (UniqueName: \"kubernetes.io/projected/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-kube-api-access-sz2d4\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:06.925220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.925224    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:06.925456 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:06.925232    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:07.460916 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.460885    2578 generic.go:356] "Generic (PLEG): container finished" podID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" containerID="49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d" exitCode=0
Sep 15 21:51:07.461102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.460929    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"
Sep 15 21:51:07.461102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.460967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" event={"ID":"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a","Type":"ContainerDied","Data":"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"}
Sep 15 21:51:07.461102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.461023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck" event={"ID":"5b5aa8b5-5d69-46a7-952e-2eb1e00f621a","Type":"ContainerDied","Data":"7ecd6f8dc46c22dad7747ec380d6c260f40168afe53689b2c6f03c647103b5ee"}
Sep 15 21:51:07.461102 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.461049    2578 scope.go:117] "RemoveContainer" containerID="49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"
Sep 15 21:51:07.470780 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.470763    2578 scope.go:117] "RemoveContainer" containerID="49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"
Sep 15 21:51:07.471058 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:51:07.471039    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d\": container with ID starting with 49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d not found: ID does not exist" containerID="49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"
Sep 15 21:51:07.471103 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.471067    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d"} err="failed to get container status \"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d\": rpc error: code = NotFound desc = could not find container \"49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d\": container with ID starting with 49ba801b6f953c3487452db8a3cf2fd27362cf480f43a0df5bbc18d13282ae8d not found: ID does not exist"
Sep 15 21:51:07.483799 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.483772    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:51:07.488778 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:07.488755    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b0ba80-5dc4bc4b77-dkkck"]
Sep 15 21:51:08.078532 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:08.078492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" path="/var/lib/kubelet/pods/5b5aa8b5-5d69-46a7-952e-2eb1e00f621a/volumes"
Sep 15 21:51:13.439647 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:13.439608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:51:13.440188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:13.439873    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy" containerID="cri-o://1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab" gracePeriod=30
Sep 15 21:51:18.689409 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.689381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:51:18.740860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.740860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.740860 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740852    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.740945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741033    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741138 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjcsr\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr\") pod \"a6789f62-8901-4877-8a01-22b4b991188d\" (UID: \"a6789f62-8901-4877-8a01-22b4b991188d\") "
Sep 15 21:51:18.741474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:18.741474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:18.741474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:18.741615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741552    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.741615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741571    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6789f62-8901-4877-8a01-22b4b991188d-istiod-ca-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.741615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741583    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-workload-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.741615 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741596    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-data\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.741804 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.741755    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:18.743323 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.743298    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:51:18.743620 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.743594    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:18.743752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.743597    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr" (OuterVolumeSpecName: "kube-api-access-gjcsr") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "kube-api-access-gjcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:18.743752 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.743731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6789f62-8901-4877-8a01-22b4b991188d" (UID: "a6789f62-8901-4877-8a01-22b4b991188d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:51:18.842807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.842769    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6789f62-8901-4877-8a01-22b4b991188d-istio-podinfo\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.842807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.842801    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-istio-token\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.842807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.842810    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjcsr\" (UniqueName: \"kubernetes.io/projected/a6789f62-8901-4877-8a01-22b4b991188d-kube-api-access-gjcsr\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.842807 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.842820    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-istio-envoy\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:18.843084 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:18.842828    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6789f62-8901-4877-8a01-22b4b991188d-credential-socket\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:19.523989 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.523949    2578 generic.go:356] "Generic (PLEG): container finished" podID="a6789f62-8901-4877-8a01-22b4b991188d" containerID="1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab" exitCode=0
Sep 15 21:51:19.524168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.524013    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"
Sep 15 21:51:19.524168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.524026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerDied","Data":"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"}
Sep 15 21:51:19.524168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.524108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh" event={"ID":"a6789f62-8901-4877-8a01-22b4b991188d","Type":"ContainerDied","Data":"3f56332536d3303494c9f3b2cef24e4d0b032f71c767f9210b2e477d35331ad4"}
Sep 15 21:51:19.524168 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.524133    2578 scope.go:117] "RemoveContainer" containerID="1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"
Sep 15 21:51:19.533786 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.533764    2578 scope.go:117] "RemoveContainer" containerID="f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"
Sep 15 21:51:19.542036 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.542016    2578 scope.go:117] "RemoveContainer" containerID="1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"
Sep 15 21:51:19.542303 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:51:19.542283    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab\": container with ID starting with 1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab not found: ID does not exist" containerID="1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"
Sep 15 21:51:19.542357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.542314    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab"} err="failed to get container status \"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab\": rpc error: code = NotFound desc = could not find container \"1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab\": container with ID starting with 1d8f1b0d5225ea49cab366f28b7ab6a67e756501196844575100704b92757dab not found: ID does not exist"
Sep 15 21:51:19.542357 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.542333    2578 scope.go:117] "RemoveContainer" containerID="f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"
Sep 15 21:51:19.542597 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:51:19.542564    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d\": container with ID starting with f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d not found: ID does not exist" containerID="f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"
Sep 15 21:51:19.542652 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.542605    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d"} err="failed to get container status \"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d\": rpc error: code = NotFound desc = could not find container \"f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d\": container with ID starting with f593b2d00b484df7edec66433cfcca53de505ce341a4be7212026a0fac58b67d not found: ID does not exist"
Sep 15 21:51:19.551421 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.551394    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:51:19.559838 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:19.559812    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0ba80-openshift-default-cdcd48ccd-ml5qh"]
Sep 15 21:51:20.075644 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:20.075608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6789f62-8901-4877-8a01-22b4b991188d" path="/var/lib/kubelet/pods/a6789f62-8901-4877-8a01-22b4b991188d/volumes"
Sep 15 21:51:24.531217 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.531182    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:51:24.531665 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.531453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" podUID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" containerName="payload-pre-processing" containerID="cri-o://c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4" gracePeriod=30
Sep 15 21:51:24.786736 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.786656    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:51:24.904433 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.904402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume\") pod \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") "
Sep 15 21:51:24.904592 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.904529    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pzkp\" (UniqueName: \"kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp\") pod \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\" (UID: \"a51f7c3f-379e-49a8-8dd5-402573ea6c4b\") "
Sep 15 21:51:24.904842 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.904819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a51f7c3f-379e-49a8-8dd5-402573ea6c4b" (UID: "a51f7c3f-379e-49a8-8dd5-402573ea6c4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:24.906554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:24.906532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp" (OuterVolumeSpecName: "kube-api-access-8pzkp") pod "a51f7c3f-379e-49a8-8dd5-402573ea6c4b" (UID: "a51f7c3f-379e-49a8-8dd5-402573ea6c4b"). InnerVolumeSpecName "kube-api-access-8pzkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:25.005797 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.005755    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:25.005797 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.005787    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pzkp\" (UniqueName: \"kubernetes.io/projected/a51f7c3f-379e-49a8-8dd5-402573ea6c4b-kube-api-access-8pzkp\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:25.558645 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.558606    2578 generic.go:356] "Generic (PLEG): container finished" podID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" containerID="c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4" exitCode=0
Sep 15 21:51:25.558645 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.558649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" event={"ID":"a51f7c3f-379e-49a8-8dd5-402573ea6c4b","Type":"ContainerDied","Data":"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"}
Sep 15 21:51:25.559230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.558655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"
Sep 15 21:51:25.559230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.558698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h" event={"ID":"a51f7c3f-379e-49a8-8dd5-402573ea6c4b","Type":"ContainerDied","Data":"c4878dc431d47ba38c017ecae5b16d97b20291dff25a7771b68f769827f1e609"}
Sep 15 21:51:25.559230 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.558741    2578 scope.go:117] "RemoveContainer" containerID="c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"
Sep 15 21:51:25.568383 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.568364    2578 scope.go:117] "RemoveContainer" containerID="c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"
Sep 15 21:51:25.568654 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:51:25.568638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4\": container with ID starting with c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4 not found: ID does not exist" containerID="c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"
Sep 15 21:51:25.568697 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.568662    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4"} err="failed to get container status \"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4\": rpc error: code = NotFound desc = could not find container \"c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4\": container with ID starting with c8514b873730bbbd8db4e0ef126bc220ab876d58155dea700cb8d118c41e92a4 not found: ID does not exist"
Sep 15 21:51:25.580116 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.580088    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:51:25.583421 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:25.583395    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-59e243-7b564f55d6-dpp9h"]
Sep 15 21:51:26.076083 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:26.076043    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" path="/var/lib/kubelet/pods/a51f7c3f-379e-49a8-8dd5-402573ea6c4b/volumes"
Sep 15 21:51:42.388386 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.388352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:51:42.388972 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.388566    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" podUID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" containerName="payload-processing" containerID="cri-o://4516efe1dc092ad21d3cb5b34ae0d9c1b68535a41676aa7ff20bd0bc35e8ba50" gracePeriod=30
Sep 15 21:51:42.400384 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.400355    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:51:42.400582 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.400558    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" podUID="b9670021-907d-4628-a067-992e8367063a" containerName="payload-pre-processing" containerID="cri-o://cac6da06fb79e59ae37ff4f3812ae45ed0e5daa5af1d4eccd2c3b214cb2aaa5e" gracePeriod=30
Sep 15 21:51:42.651246 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.651159    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" containerID="4516efe1dc092ad21d3cb5b34ae0d9c1b68535a41676aa7ff20bd0bc35e8ba50" exitCode=0
Sep 15 21:51:42.651246 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.651194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" event={"ID":"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88","Type":"ContainerDied","Data":"4516efe1dc092ad21d3cb5b34ae0d9c1b68535a41676aa7ff20bd0bc35e8ba50"}
Sep 15 21:51:42.651455 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.651263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p" event={"ID":"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88","Type":"ContainerDied","Data":"8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba"}
Sep 15 21:51:42.651455 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.651290    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8249f69cbab845a20a9fcb1933baca6620235402a31b4c827fbdf91f7cdb7aba"
Sep 15 21:51:42.653111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.653082    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9670021-907d-4628-a067-992e8367063a" containerID="cac6da06fb79e59ae37ff4f3812ae45ed0e5daa5af1d4eccd2c3b214cb2aaa5e" exitCode=0
Sep 15 21:51:42.653271 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.653111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" event={"ID":"b9670021-907d-4628-a067-992e8367063a","Type":"ContainerDied","Data":"cac6da06fb79e59ae37ff4f3812ae45ed0e5daa5af1d4eccd2c3b214cb2aaa5e"}
Sep 15 21:51:42.655509 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.655489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:51:42.670981 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.670955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz98r\" (UniqueName: \"kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r\") pod \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") "
Sep 15 21:51:42.671107 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.671052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume\") pod \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\" (UID: \"f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88\") "
Sep 15 21:51:42.671391 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.671370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" (UID: "f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:42.673139 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.673120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r" (OuterVolumeSpecName: "kube-api-access-dz98r") pod "f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" (UID: "f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88"). InnerVolumeSpecName "kube-api-access-dz98r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:42.687241 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.687213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:51:42.772192 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.772159    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume\") pod \"b9670021-907d-4628-a067-992e8367063a\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") "
Sep 15 21:51:42.772375 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.772205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l42pp\" (UniqueName: \"kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp\") pod \"b9670021-907d-4628-a067-992e8367063a\" (UID: \"b9670021-907d-4628-a067-992e8367063a\") "
Sep 15 21:51:42.772421 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.772381    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz98r\" (UniqueName: \"kubernetes.io/projected/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-kube-api-access-dz98r\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:42.772421 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.772394    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:42.772660 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.772634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9670021-907d-4628-a067-992e8367063a" (UID: "b9670021-907d-4628-a067-992e8367063a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:51:42.774378 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.774348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp" (OuterVolumeSpecName: "kube-api-access-l42pp") pod "b9670021-907d-4628-a067-992e8367063a" (UID: "b9670021-907d-4628-a067-992e8367063a"). InnerVolumeSpecName "kube-api-access-l42pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:42.873296 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.873257    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9670021-907d-4628-a067-992e8367063a-plugins-config-volume\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:42.873296 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:42.873288    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l42pp\" (UniqueName: \"kubernetes.io/projected/b9670021-907d-4628-a067-992e8367063a-kube-api-access-l42pp\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:43.659928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.659791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"
Sep 15 21:51:43.659928 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.659845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9" event={"ID":"b9670021-907d-4628-a067-992e8367063a","Type":"ContainerDied","Data":"90ad368d78680c505c402fb97c027a72f018a2268c65418f1ab08ff9a4395581"}
Sep 15 21:51:43.660560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.659932    2578 scope.go:117] "RemoveContainer" containerID="cac6da06fb79e59ae37ff4f3812ae45ed0e5daa5af1d4eccd2c3b214cb2aaa5e"
Sep 15 21:51:43.660560 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.660017    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"
Sep 15 21:51:43.685670 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.685637    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:51:43.689026 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.688999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-38a3a3-55f8c6d678-m77n9"]
Sep 15 21:51:43.704641 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.704609    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:51:43.709497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:43.709466    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-38a3a3-6c78889767-hcp7p"]
Sep 15 21:51:44.076090 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:44.076058    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9670021-907d-4628-a067-992e8367063a" path="/var/lib/kubelet/pods/b9670021-907d-4628-a067-992e8367063a/volumes"
Sep 15 21:51:44.076450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:44.076437    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" path="/var/lib/kubelet/pods/f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88/volumes"
Sep 15 21:51:52.461366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:52.461326    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:51:52.461788 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:52.461532    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bd4b7887f-qskxh" podUID="276a075a-a2bf-47c3-bc81-d1375f92993e" containerName="manager" containerID="cri-o://9e5147ddd53f0289014f5b89eae128f13a497f842331c3eb11d97277529c7258" gracePeriod=30
Sep 15 21:51:52.711120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:52.711081    2578 generic.go:356] "Generic (PLEG): container finished" podID="276a075a-a2bf-47c3-bc81-d1375f92993e" containerID="9e5147ddd53f0289014f5b89eae128f13a497f842331c3eb11d97277529c7258" exitCode=0
Sep 15 21:51:52.711299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:52.711154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qskxh" event={"ID":"276a075a-a2bf-47c3-bc81-d1375f92993e","Type":"ContainerDied","Data":"9e5147ddd53f0289014f5b89eae128f13a497f842331c3eb11d97277529c7258"}
Sep 15 21:51:53.216843 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.216817    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:51:53.282617 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.282586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l858c\" (UniqueName: \"kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c\") pod \"276a075a-a2bf-47c3-bc81-d1375f92993e\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") "
Sep 15 21:51:53.282836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.282658    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs\") pod \"276a075a-a2bf-47c3-bc81-d1375f92993e\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") "
Sep 15 21:51:53.282836 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.282748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert\") pod \"276a075a-a2bf-47c3-bc81-d1375f92993e\" (UID: \"276a075a-a2bf-47c3-bc81-d1375f92993e\") "
Sep 15 21:51:53.284738 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.284689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c" (OuterVolumeSpecName: "kube-api-access-l858c") pod "276a075a-a2bf-47c3-bc81-d1375f92993e" (UID: "276a075a-a2bf-47c3-bc81-d1375f92993e"). InnerVolumeSpecName "kube-api-access-l858c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:51:53.284876 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.284850    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "276a075a-a2bf-47c3-bc81-d1375f92993e" (UID: "276a075a-a2bf-47c3-bc81-d1375f92993e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:53.284955 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.284933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "276a075a-a2bf-47c3-bc81-d1375f92993e" (UID: "276a075a-a2bf-47c3-bc81-d1375f92993e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:51:53.383500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.383411    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l858c\" (UniqueName: \"kubernetes.io/projected/276a075a-a2bf-47c3-bc81-d1375f92993e-kube-api-access-l858c\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:53.383500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.383442    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:53.383500 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.383451    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/276a075a-a2bf-47c3-bc81-d1375f92993e-webhook-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:51:53.717554 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.717529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qskxh"
Sep 15 21:51:53.718002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.717522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qskxh" event={"ID":"276a075a-a2bf-47c3-bc81-d1375f92993e","Type":"ContainerDied","Data":"c477000d15120f08ab39035caf039b1b33b6f7bf813fd7dc34c480342be6b606"}
Sep 15 21:51:53.718002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.717669    2578 scope.go:117] "RemoveContainer" containerID="9e5147ddd53f0289014f5b89eae128f13a497f842331c3eb11d97277529c7258"
Sep 15 21:51:53.739238 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.739211    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:51:53.745535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:53.745513    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qskxh"]
Sep 15 21:51:54.075493 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:51:54.075409    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276a075a-a2bf-47c3-bc81-d1375f92993e" path="/var/lib/kubelet/pods/276a075a-a2bf-47c3-bc81-d1375f92993e/volumes"
Sep 15 21:52:12.415453 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:12.415370    2578 scope.go:117] "RemoveContainer" containerID="4516efe1dc092ad21d3cb5b34ae0d9c1b68535a41676aa7ff20bd0bc35e8ba50"
Sep 15 21:52:54.526821 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.526782    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:52:54.527308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527280    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" containerName="payload-processing"
Sep 15 21:52:54.527308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527302    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" containerName="payload-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527313    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9670021-907d-4628-a067-992e8367063a" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527319    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9670021-907d-4628-a067-992e8367063a" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527326    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" containerName="maas-api"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527337    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" containerName="maas-api"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527345    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52682831-e206-4fc0-ae3f-b631fa6d12fb" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527351    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52682831-e206-4fc0-ae3f-b631fa6d12fb" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527359    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527365    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" containerName="payload-pre-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527373    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527378    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527383    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527389    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527400    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" containerName="payload-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527405    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" containerName="payload-processing"
Sep 15 21:52:54.527405 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae06ab8-b13a-46d1-934c-133febe03e54" containerName="maas-api"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae06ab8-b13a-46d1-934c-133febe03e54" containerName="maas-api"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527429    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276a075a-a2bf-47c3-bc81-d1375f92993e" containerName="manager"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527435    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="276a075a-a2bf-47c3-bc81-d1375f92993e" containerName="manager"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527442    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" containerName="payload-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527447    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" containerName="payload-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527514    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5aa8b5-5d69-46a7-952e-2eb1e00f621a" containerName="maas-api"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527521    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52682831-e206-4fc0-ae3f-b631fa6d12fb" containerName="payload-pre-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527527    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="276a075a-a2bf-47c3-bc81-d1375f92993e" containerName="manager"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527534    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527538    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="131dcf3e-fe7f-4f0a-bff9-9592d73f4002" containerName="payload-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527546    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae06ab8-b13a-46d1-934c-133febe03e54" containerName="maas-api"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527552    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6789f62-8901-4877-8a01-22b4b991188d" containerName="istio-proxy"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527557    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fc4ca5-bf8a-4d5f-a1ae-945962a26d88" containerName="payload-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527563    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51f7c3f-379e-49a8-8dd5-402573ea6c4b" containerName="payload-pre-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527568    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9670021-907d-4628-a067-992e8367063a" containerName="payload-pre-processing"
Sep 15 21:52:54.527941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.527573    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ebf170-bcfe-45db-a235-fa52d26b2b5c" containerName="payload-processing"
Sep 15 21:52:54.530556 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.530539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.532663 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.532633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:52:54.532830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.532636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vsdzf\""
Sep 15 21:52:54.532830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.532637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:52:54.532830 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.532685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:52:54.538756 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.538734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:52:54.572882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.572848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.573025 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.572970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.573025 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.573007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72d4q\" (UniqueName: \"kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.674264 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.674217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.674474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.674280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d4q\" (UniqueName: \"kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.674474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.674315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.676757 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.676709    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.676875 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.676834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.682037 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.682019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72d4q\" (UniqueName: \"kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q\") pod \"maas-controller-bd4b7887f-mcp52\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") " pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:54.842816 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:54.842735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:55.115221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:55.115063    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:52:55.117588 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:52:55.117559    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136b3535_0761_443f_9e57_acc28a61b13f.slice/crio-2ace5c55e030be9ce12f292e5d13c4e2b5a339148a5151e0124f3cbe4b73f291 WatchSource:0}: Error finding container 2ace5c55e030be9ce12f292e5d13c4e2b5a339148a5151e0124f3cbe4b73f291: Status 404 returned error can't find the container with id 2ace5c55e030be9ce12f292e5d13c4e2b5a339148a5151e0124f3cbe4b73f291
Sep 15 21:52:55.119053 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:55.119031    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:52:56.024286 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:56.024247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-mcp52" event={"ID":"136b3535-0761-443f-9e57-acc28a61b13f","Type":"ContainerStarted","Data":"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"}
Sep 15 21:52:56.024763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:56.024309    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-mcp52" event={"ID":"136b3535-0761-443f-9e57-acc28a61b13f","Type":"ContainerStarted","Data":"2ace5c55e030be9ce12f292e5d13c4e2b5a339148a5151e0124f3cbe4b73f291"}
Sep 15 21:52:56.024763 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:56.024454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:52:56.041673 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:52:56.041619    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd4b7887f-mcp52" podStartSLOduration=1.649747519 podStartE2EDuration="2.041601274s" totalImagesPullingTime="391.853755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:56.040751011 +0000 UTC m=+2204.441724246" watchObservedRunningTime="2026-09-15 21:52:56.041601274 +0000 UTC m=+2204.442574520"
Sep 15 21:53:07.031384 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:07.031349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:53:14.213449 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.213409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:53:14.213994 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.213636    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-bd4b7887f-mcp52" podUID="136b3535-0761-443f-9e57-acc28a61b13f" containerName="manager" containerID="cri-o://0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8" gracePeriod=30
Sep 15 21:53:14.461393 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.461364    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:53:14.578188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.578101    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert\") pod \"136b3535-0761-443f-9e57-acc28a61b13f\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") "
Sep 15 21:53:14.578188 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.578170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs\") pod \"136b3535-0761-443f-9e57-acc28a61b13f\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") "
Sep 15 21:53:14.578366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.578193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72d4q\" (UniqueName: \"kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q\") pod \"136b3535-0761-443f-9e57-acc28a61b13f\" (UID: \"136b3535-0761-443f-9e57-acc28a61b13f\") "
Sep 15 21:53:14.580257 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.580217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "136b3535-0761-443f-9e57-acc28a61b13f" (UID: "136b3535-0761-443f-9e57-acc28a61b13f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:53:14.580402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.580259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q" (OuterVolumeSpecName: "kube-api-access-72d4q") pod "136b3535-0761-443f-9e57-acc28a61b13f" (UID: "136b3535-0761-443f-9e57-acc28a61b13f"). InnerVolumeSpecName "kube-api-access-72d4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:14.580402 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.580284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "136b3535-0761-443f-9e57-acc28a61b13f" (UID: "136b3535-0761-443f-9e57-acc28a61b13f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:53:14.679179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.679137    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-webhook-cert\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:53:14.679179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.679169    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136b3535-0761-443f-9e57-acc28a61b13f-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:53:14.679179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:14.679182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72d4q\" (UniqueName: \"kubernetes.io/projected/136b3535-0761-443f-9e57-acc28a61b13f-kube-api-access-72d4q\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:53:15.124981 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.124941    2578 generic.go:356] "Generic (PLEG): container finished" podID="136b3535-0761-443f-9e57-acc28a61b13f" containerID="0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8" exitCode=0
Sep 15 21:53:15.125187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.124987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-mcp52"
Sep 15 21:53:15.125187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.125002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-mcp52" event={"ID":"136b3535-0761-443f-9e57-acc28a61b13f","Type":"ContainerDied","Data":"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"}
Sep 15 21:53:15.125187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.125053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-mcp52" event={"ID":"136b3535-0761-443f-9e57-acc28a61b13f","Type":"ContainerDied","Data":"2ace5c55e030be9ce12f292e5d13c4e2b5a339148a5151e0124f3cbe4b73f291"}
Sep 15 21:53:15.125187 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.125080    2578 scope.go:117] "RemoveContainer" containerID="0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"
Sep 15 21:53:15.135438 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.135416    2578 scope.go:117] "RemoveContainer" containerID="0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"
Sep 15 21:53:15.135828 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:53:15.135796    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8\": container with ID starting with 0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8 not found: ID does not exist" containerID="0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"
Sep 15 21:53:15.135961 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.135835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8"} err="failed to get container status \"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8\": rpc error: code = NotFound desc = could not find container \"0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8\": container with ID starting with 0e28c9ed8ad2e2b37d434a73207479caeed6ade38206a5303e23fc30caddbaf8 not found: ID does not exist"
Sep 15 21:53:15.148858 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.148823    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:53:15.157794 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.157750    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-mcp52"]
Sep 15 21:53:15.923659 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.923620    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qhsnm"]
Sep 15 21:53:15.924073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.924025    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136b3535-0761-443f-9e57-acc28a61b13f" containerName="manager"
Sep 15 21:53:15.924073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.924037    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="136b3535-0761-443f-9e57-acc28a61b13f" containerName="manager"
Sep 15 21:53:15.924148 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.924123    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="136b3535-0761-443f-9e57-acc28a61b13f" containerName="manager"
Sep 15 21:53:15.928535 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.928516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:15.930631 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.930606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:53:15.930951 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.930927    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:53:15.931025 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.930929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vsdzf\""
Sep 15 21:53:15.931025 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.930961    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:53:15.941792 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:15.941759    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qhsnm"]
Sep 15 21:53:16.075782 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.075748    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136b3535-0761-443f-9e57-acc28a61b13f" path="/var/lib/kubelet/pods/136b3535-0761-443f-9e57-acc28a61b13f/volumes"
Sep 15 21:53:16.094703 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.094672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-metrics-certs\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.094853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.094706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-webhook-cert\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.094853 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.094757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clmw\" (UniqueName: \"kubernetes.io/projected/40d99f10-e736-4ba1-a2d9-173941e51abd-kube-api-access-2clmw\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.195731 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.195684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-metrics-certs\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.195892 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.195751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-webhook-cert\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.195892 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.195777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clmw\" (UniqueName: \"kubernetes.io/projected/40d99f10-e736-4ba1-a2d9-173941e51abd-kube-api-access-2clmw\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.197960 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.197939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-webhook-cert\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.198069 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.197993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40d99f10-e736-4ba1-a2d9-173941e51abd-metrics-certs\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.202364 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.202346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clmw\" (UniqueName: \"kubernetes.io/projected/40d99f10-e736-4ba1-a2d9-173941e51abd-kube-api-access-2clmw\") pod \"maas-controller-bd4b7887f-qhsnm\" (UID: \"40d99f10-e736-4ba1-a2d9-173941e51abd\") " pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.247411 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.247380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:16.520221 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:16.520194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bd4b7887f-qhsnm"]
Sep 15 21:53:16.522262 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:53:16.522234    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d99f10_e736_4ba1_a2d9_173941e51abd.slice/crio-3aa0a77b8600c7a95ced2931a4ffeaaef5b2be14ffd23b1d8f1a06b383a4d98f WatchSource:0}: Error finding container 3aa0a77b8600c7a95ced2931a4ffeaaef5b2be14ffd23b1d8f1a06b383a4d98f: Status 404 returned error can't find the container with id 3aa0a77b8600c7a95ced2931a4ffeaaef5b2be14ffd23b1d8f1a06b383a4d98f
Sep 15 21:53:17.137754 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:17.137650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qhsnm" event={"ID":"40d99f10-e736-4ba1-a2d9-173941e51abd","Type":"ContainerStarted","Data":"910d157a576a7a1996081468c88be60fe98376ee225e9ab2b5ae9fc7985b02e3"}
Sep 15 21:53:17.137754 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:17.137710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bd4b7887f-qhsnm" event={"ID":"40d99f10-e736-4ba1-a2d9-173941e51abd","Type":"ContainerStarted","Data":"3aa0a77b8600c7a95ced2931a4ffeaaef5b2be14ffd23b1d8f1a06b383a4d98f"}
Sep 15 21:53:17.138179 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:17.137761    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:53:17.155974 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:17.155920    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-bd4b7887f-qhsnm" podStartSLOduration=1.82989901 podStartE2EDuration="2.155903107s" totalImagesPullingTime="326.004097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:17.153866405 +0000 UTC m=+2225.554839652" watchObservedRunningTime="2026-09-15 21:53:17.155903107 +0000 UTC m=+2225.556876353"
Sep 15 21:53:28.146142 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:53:28.146109    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bd4b7887f-qhsnm"
Sep 15 21:56:36.501542 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.501438    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:56:36.505789 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.505766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.507863 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.507836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f2776979-kserve-self-signed-certs\""
Sep 15 21:56:36.516146 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.516115    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:56:36.545300 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.545454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.545454 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4x7\" (UniqueName: \"kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.545543 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.545589 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.545631 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.545595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647162 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4x7\" (UniqueName: \"kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647488 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647611 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647748 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.647835 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.647787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.650017 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.649996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.650401 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.650378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.656012 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.655989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4x7\" (UniqueName: \"kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7\") pod \"e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") " pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:36.817050 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:36.816955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:37.088709 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:37.088682    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:56:37.091009 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:56:37.090982    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ee519d4_e98d_451c_b63f_77f27d836e32.slice/crio-b4e0d4de949fc22949c1d8db320c610113e3fd9f3413d60d1038889f3a887f38 WatchSource:0}: Error finding container b4e0d4de949fc22949c1d8db320c610113e3fd9f3413d60d1038889f3a887f38: Status 404 returned error can't find the container with id b4e0d4de949fc22949c1d8db320c610113e3fd9f3413d60d1038889f3a887f38
Sep 15 21:56:37.138941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:37.138906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" event={"ID":"4ee519d4-e98d-451c-b63f-77f27d836e32","Type":"ContainerStarted","Data":"b4e0d4de949fc22949c1d8db320c610113e3fd9f3413d60d1038889f3a887f38"}
Sep 15 21:56:38.146111 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:38.146075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" event={"ID":"4ee519d4-e98d-451c-b63f-77f27d836e32","Type":"ContainerStarted","Data":"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"}
Sep 15 21:56:38.164621 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:38.164562    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" podStartSLOduration=2.164545979 podStartE2EDuration="2.164545979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:56:38.163824072 +0000 UTC m=+2426.564797313" watchObservedRunningTime="2026-09-15 21:56:38.164545979 +0000 UTC m=+2426.565519257"
Sep 15 21:56:46.817624 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:46.817588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:46.818196 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:46.817661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:46.830145 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:46.830120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:56:47.202969 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:56:47.202935    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:57:10.795629 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:10.795595    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:57:25.815693 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:25.815626    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" podUID="4ee519d4-e98d-451c-b63f-77f27d836e32" containerName="main" containerID="cri-o://1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b" gracePeriod=45
Sep 15 21:57:26.067457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.067393    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:57:26.229503 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn4x7\" (UniqueName: \"kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229563    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229610    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229699 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm\") pod \"4ee519d4-e98d-451c-b63f-77f27d836e32\" (UID: \"4ee519d4-e98d-451c-b63f-77f27d836e32\") "
Sep 15 21:57:26.229962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:26.229962 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home" (OuterVolumeSpecName: "home") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:26.230121 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.229989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:26.230497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.230252    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-tmp-dir\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.230497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.230280    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-model-cache\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.230497 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.230293    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-home\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.232092 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.232060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm" (OuterVolumeSpecName: "dshm") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:26.232287 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.232265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:57:26.232358 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.232286    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7" (OuterVolumeSpecName: "kube-api-access-dn4x7") pod "4ee519d4-e98d-451c-b63f-77f27d836e32" (UID: "4ee519d4-e98d-451c-b63f-77f27d836e32"). InnerVolumeSpecName "kube-api-access-dn4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:57:26.331003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.330908    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ee519d4-e98d-451c-b63f-77f27d836e32-tls-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.331003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.330942    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ee519d4-e98d-451c-b63f-77f27d836e32-dshm\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.331003 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.330951    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4x7\" (UniqueName: \"kubernetes.io/projected/4ee519d4-e98d-451c-b63f-77f27d836e32-kube-api-access-dn4x7\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:26.399567 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.399528    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ee519d4-e98d-451c-b63f-77f27d836e32" containerID="1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b" exitCode=0
Sep 15 21:57:26.399772 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.399579    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"
Sep 15 21:57:26.399772 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.399615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" event={"ID":"4ee519d4-e98d-451c-b63f-77f27d836e32","Type":"ContainerDied","Data":"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"}
Sep 15 21:57:26.399772 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.399652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv" event={"ID":"4ee519d4-e98d-451c-b63f-77f27d836e32","Type":"ContainerDied","Data":"b4e0d4de949fc22949c1d8db320c610113e3fd9f3413d60d1038889f3a887f38"}
Sep 15 21:57:26.399772 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.399671    2578 scope.go:117] "RemoveContainer" containerID="1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"
Sep 15 21:57:26.411422 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.411333    2578 scope.go:117] "RemoveContainer" containerID="1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"
Sep 15 21:57:26.411928 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:57:26.411884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b\": container with ID starting with 1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b not found: ID does not exist" containerID="1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"
Sep 15 21:57:26.412002 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.411935    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b"} err="failed to get container status \"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b\": rpc error: code = NotFound desc = could not find container \"1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b\": container with ID starting with 1066b5e9938fbf03c3bc33881bc5a54d218e65ae5a8205d5fab2287afcbff28b not found: ID does not exist"
Sep 15 21:57:26.439636 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.439606    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:57:26.442952 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:26.442930    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f2776979-kserve-74bb6bdc6c-4dtsv"]
Sep 15 21:57:28.075896 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:28.075861    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ee519d4-e98d-451c-b63f-77f27d836e32" path="/var/lib/kubelet/pods/4ee519d4-e98d-451c-b63f-77f27d836e32/volumes"
Sep 15 21:57:29.699546 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.699506    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"]
Sep 15 21:57:29.700201 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.700178    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ee519d4-e98d-451c-b63f-77f27d836e32" containerName="main"
Sep 15 21:57:29.700248 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.700206    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ee519d4-e98d-451c-b63f-77f27d836e32" containerName="main"
Sep 15 21:57:29.700303 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.700293    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ee519d4-e98d-451c-b63f-77f27d836e32" containerName="main"
Sep 15 21:57:29.705198 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.705171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.718649 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.718625    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"]
Sep 15 21:57:29.864114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.864075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-metrics-certs\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.864114 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.864115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99c6s\" (UniqueName: \"kubernetes.io/projected/edc8483a-da96-4aff-8439-9ad68443f50a-kube-api-access-99c6s\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.864340 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.864237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-maas-api-tls\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.864340 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.864272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc8483a-da96-4aff-8439-9ad68443f50a-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.965788 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.965669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-maas-api-tls\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.965950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.965843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc8483a-da96-4aff-8439-9ad68443f50a-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.965950 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.965922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-metrics-certs\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.966072 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.965950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99c6s\" (UniqueName: \"kubernetes.io/projected/edc8483a-da96-4aff-8439-9ad68443f50a-kube-api-access-99c6s\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.966491 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.966465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc8483a-da96-4aff-8439-9ad68443f50a-openshift-service-ca\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.968166 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.968144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-maas-api-tls\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.968312 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.968294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc8483a-da96-4aff-8439-9ad68443f50a-metrics-certs\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:29.974501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:29.974478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99c6s\" (UniqueName: \"kubernetes.io/projected/edc8483a-da96-4aff-8439-9ad68443f50a-kube-api-access-99c6s\") pod \"maas-api-85c7dfbd4c-4k86x\" (UID: \"edc8483a-da96-4aff-8439-9ad68443f50a\") " pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:30.015227 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:30.015192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:30.294095 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:30.294069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"]
Sep 15 21:57:30.297044 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:57:30.297006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc8483a_da96_4aff_8439_9ad68443f50a.slice/crio-0a94adf7915fe71d5cb1359a74c19f4a3b320fb5588681f63d2d76aca0242937 WatchSource:0}: Error finding container 0a94adf7915fe71d5cb1359a74c19f4a3b320fb5588681f63d2d76aca0242937: Status 404 returned error can't find the container with id 0a94adf7915fe71d5cb1359a74c19f4a3b320fb5588681f63d2d76aca0242937
Sep 15 21:57:30.422790 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:30.422751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x" event={"ID":"edc8483a-da96-4aff-8439-9ad68443f50a","Type":"ContainerStarted","Data":"0a94adf7915fe71d5cb1359a74c19f4a3b320fb5588681f63d2d76aca0242937"}
Sep 15 21:57:31.429224 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:31.429181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x" event={"ID":"edc8483a-da96-4aff-8439-9ad68443f50a","Type":"ContainerStarted","Data":"1512f8725a92d77da3c9021ed52b8ac8329ef1a2b2777c531202ef9c6be39855"}
Sep 15 21:57:31.429822 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:31.429292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:31.459382 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:31.459328    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x" podStartSLOduration=2.018770866 podStartE2EDuration="2.459313592s" totalImagesPullingTime="440.542726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:31.456641342 +0000 UTC m=+2479.857614587" watchObservedRunningTime="2026-09-15 21:57:31.459313592 +0000 UTC m=+2479.860286838"
Sep 15 21:57:37.439769 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:37.439735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85c7dfbd4c-4k86x"
Sep 15 21:57:37.574900 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:37.574870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:57:37.575076 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:37.575052    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" podUID="b75e843c-1f08-44ab-b565-68517fd7f343" containerName="maas-api" containerID="cri-o://440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d" gracePeriod=30
Sep 15 21:57:38.823341 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.823313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:57:38.855132 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.855095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") pod \"b75e843c-1f08-44ab-b565-68517fd7f343\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") "
Sep 15 21:57:38.855316 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.855179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") pod \"b75e843c-1f08-44ab-b565-68517fd7f343\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") "
Sep 15 21:57:38.855316 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.855259    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca\") pod \"b75e843c-1f08-44ab-b565-68517fd7f343\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") "
Sep 15 21:57:38.855452 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.855324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c27gf\" (UniqueName: \"kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf\") pod \"b75e843c-1f08-44ab-b565-68517fd7f343\" (UID: \"b75e843c-1f08-44ab-b565-68517fd7f343\") "
Sep 15 21:57:38.855690 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.855659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b75e843c-1f08-44ab-b565-68517fd7f343" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:57:38.857334 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.857299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf" (OuterVolumeSpecName: "kube-api-access-c27gf") pod "b75e843c-1f08-44ab-b565-68517fd7f343" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343"). InnerVolumeSpecName "kube-api-access-c27gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:57:38.857429 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.857402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b75e843c-1f08-44ab-b565-68517fd7f343" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:57:38.857474 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.857421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b75e843c-1f08-44ab-b565-68517fd7f343" (UID: "b75e843c-1f08-44ab-b565-68517fd7f343"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:57:38.956948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.956917    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-metrics-certs\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:38.956948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.956947    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e843c-1f08-44ab-b565-68517fd7f343-openshift-service-ca\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:38.957131 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.956958    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27gf\" (UniqueName: \"kubernetes.io/projected/b75e843c-1f08-44ab-b565-68517fd7f343-kube-api-access-c27gf\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:38.957131 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:38.956967    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e843c-1f08-44ab-b565-68517fd7f343-maas-api-tls\") on node \"ip-10-0-135-105.ec2.internal\" DevicePath \"\""
Sep 15 21:57:39.468527 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.468492    2578 generic.go:356] "Generic (PLEG): container finished" podID="b75e843c-1f08-44ab-b565-68517fd7f343" containerID="440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d" exitCode=0
Sep 15 21:57:39.468706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.468540    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"
Sep 15 21:57:39.468706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.468581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" event={"ID":"b75e843c-1f08-44ab-b565-68517fd7f343","Type":"ContainerDied","Data":"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"}
Sep 15 21:57:39.468706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.468635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6" event={"ID":"b75e843c-1f08-44ab-b565-68517fd7f343","Type":"ContainerDied","Data":"71274eac7e4b14eb7d92d47121af020fc59285d404212bdc7571901e6f31682b"}
Sep 15 21:57:39.468706 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.468659    2578 scope.go:117] "RemoveContainer" containerID="440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"
Sep 15 21:57:39.478081 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.478065    2578 scope.go:117] "RemoveContainer" containerID="440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"
Sep 15 21:57:39.478316 ip-10-0-135-105 kubenswrapper[2578]: E0915 21:57:39.478300    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d\": container with ID starting with 440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d not found: ID does not exist" containerID="440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"
Sep 15 21:57:39.478363 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.478323    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d"} err="failed to get container status \"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d\": rpc error: code = NotFound desc = could not find container \"440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d\": container with ID starting with 440c5359628144fd0376ec54cc5bc5bbb93f46e5b2a5fd27952dc59efe6dab1d not found: ID does not exist"
Sep 15 21:57:39.501141 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.501116    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:57:39.511564 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:39.511539    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-97c9c8fdc-qm5h6"]
Sep 15 21:57:40.075959 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:40.075926    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75e843c-1f08-44ab-b565-68517fd7f343" path="/var/lib/kubelet/pods/b75e843c-1f08-44ab-b565-68517fd7f343/volumes"
Sep 15 21:57:59.750165 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:57:59.750134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f847c588-ws6zl_b80d04e9-56f1-49c7-801c-adfa3daad9f2/authorino/0.log"
Sep 15 21:58:05.876049 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:05.875964    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd4b7887f-qhsnm_40d99f10-e736-4ba1-a2d9-173941e51abd/manager/0.log"
Sep 15 21:58:06.270305 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:06.270277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-f6zdk_27835161-a6a8-4f98-bc79-8825748e6b66/copy-manifests/0.log"
Sep 15 21:58:06.283815 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:06.283786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-f6zdk_27835161-a6a8-4f98-bc79-8825748e6b66/manager/0.log"
Sep 15 21:58:06.930577 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:06.930545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85c7dfbd4c-4k86x_edc8483a-da96-4aff-8439-9ad68443f50a/maas-api/0.log"
Sep 15 21:58:07.279047 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:07.279015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825145-pfw2j_cbe6cebe-b991-4efc-83f1-9aad9c43ae92/cleanup/0.log"
Sep 15 21:58:08.177083 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:08.177052    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f847c588-ws6zl_b80d04e9-56f1-49c7-801c-adfa3daad9f2/authorino/0.log"
Sep 15 21:58:08.563787 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:08.563759    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wnrnd_13de072d-f42d-46e1-a663-02fbd6939e8c/kuadrant-console-plugin/0.log"
Sep 15 21:58:09.677980 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:09.677951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-z4mdg_23a497b4-8e8b-4bf3-a03f-6af1bdd58259/kube-auth-proxy/0.log"
Sep 15 21:58:09.942759 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:09.942724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rfw5s_13951561-032d-4591-88e5-ea2e0befd373/payload-pre-processing/0.log"
Sep 15 21:58:10.565548 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:10.565517    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-n2wld_0e1e84e3-657d-45d4-a2ab-ef70a38adb48/main/0.log"
Sep 15 21:58:11.060982 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:11.060948    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fb5hqj_ba6512e4-09f8-4e73-aef6-0051114c32b3/main/0.log"
Sep 15 21:58:17.064937 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.064897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7p98/must-gather-zfjh8"]
Sep 15 21:58:17.065363 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.065348    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75e843c-1f08-44ab-b565-68517fd7f343" containerName="maas-api"
Sep 15 21:58:17.065409 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.065364    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75e843c-1f08-44ab-b565-68517fd7f343" containerName="maas-api"
Sep 15 21:58:17.065450 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.065429    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75e843c-1f08-44ab-b565-68517fd7f343" containerName="maas-api"
Sep 15 21:58:17.069993 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.069975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.073347 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.073315    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"openshift-service-ca.crt\""
Sep 15 21:58:17.073489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.073353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j7p98\"/\"default-dockercfg-87jck\""
Sep 15 21:58:17.073489 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.073375    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j7p98\"/\"kube-root-ca.crt\""
Sep 15 21:58:17.098593 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.098565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/must-gather-zfjh8"]
Sep 15 21:58:17.227591 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.227554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f142dec9-c554-4317-bdd3-775072545e2f-must-gather-output\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.227815 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.227677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvvx\" (UniqueName: \"kubernetes.io/projected/f142dec9-c554-4317-bdd3-775072545e2f-kube-api-access-jwvvx\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.328602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.328512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvvx\" (UniqueName: \"kubernetes.io/projected/f142dec9-c554-4317-bdd3-775072545e2f-kube-api-access-jwvvx\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.328602 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.328581    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f142dec9-c554-4317-bdd3-775072545e2f-must-gather-output\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.328992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.328963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f142dec9-c554-4317-bdd3-775072545e2f-must-gather-output\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.342770 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.342737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvvx\" (UniqueName: \"kubernetes.io/projected/f142dec9-c554-4317-bdd3-775072545e2f-kube-api-access-jwvvx\") pod \"must-gather-zfjh8\" (UID: \"f142dec9-c554-4317-bdd3-775072545e2f\") " pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.378838 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.378804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/must-gather-zfjh8"
Sep 15 21:58:17.657820 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.657792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/must-gather-zfjh8"]
Sep 15 21:58:17.660034 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:58:17.660006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf142dec9_c554_4317_bdd3_775072545e2f.slice/crio-097d2afbd39c81b72762e388631db13cf8e16e30727ba8bba98ec6dbc73e607b WatchSource:0}: Error finding container 097d2afbd39c81b72762e388631db13cf8e16e30727ba8bba98ec6dbc73e607b: Status 404 returned error can't find the container with id 097d2afbd39c81b72762e388631db13cf8e16e30727ba8bba98ec6dbc73e607b
Sep 15 21:58:17.662069 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:17.662050    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:58:18.661875 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:18.661843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/must-gather-zfjh8" event={"ID":"f142dec9-c554-4317-bdd3-775072545e2f","Type":"ContainerStarted","Data":"097d2afbd39c81b72762e388631db13cf8e16e30727ba8bba98ec6dbc73e607b"}
Sep 15 21:58:19.671071 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:19.671035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/must-gather-zfjh8" event={"ID":"f142dec9-c554-4317-bdd3-775072545e2f","Type":"ContainerStarted","Data":"2624ecd24e7a18640a8d39730aeba2e488a04c7a9969be190128887858e88ae6"}
Sep 15 21:58:19.671071 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:19.671080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/must-gather-zfjh8" event={"ID":"f142dec9-c554-4317-bdd3-775072545e2f","Type":"ContainerStarted","Data":"5330defda5cb37d9c044eecb356401d706dcd864b59795d497355366a43205c5"}
Sep 15 21:58:19.711596 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:19.711528    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7p98/must-gather-zfjh8" podStartSLOduration=1.787199268 podStartE2EDuration="2.7115006s" totalImagesPullingTime="924.301332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:19.70998544 +0000 UTC m=+2528.110958687" watchObservedRunningTime="2026-09-15 21:58:19.7115006 +0000 UTC m=+2528.112473846"
Sep 15 21:58:20.976882 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:20.976849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z4tbr_7aa081ed-af1f-44c1-b4ba-25ae8c2c2b94/global-pull-secret-syncer/0.log"
Sep 15 21:58:21.284990 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:21.284901    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wltw9_b312a59f-32c2-4483-8147-28b5e2f0df15/konnectivity-agent/0.log"
Sep 15 21:58:21.418608 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:21.418577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-105.ec2.internal_54b69627995a64131f3948c3f8331567/haproxy/0.log"
Sep 15 21:58:24.898620 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:24.898581    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85f847c588-ws6zl_b80d04e9-56f1-49c7-801c-adfa3daad9f2/authorino/0.log"
Sep 15 21:58:25.255190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:25.255154    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wnrnd_13de072d-f42d-46e1-a663-02fbd6939e8c/kuadrant-console-plugin/0.log"
Sep 15 21:58:27.429948 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.429099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/alertmanager/0.log"
Sep 15 21:58:27.471337 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.471301    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/config-reloader/0.log"
Sep 15 21:58:27.513089 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.513042    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/kube-rbac-proxy-web/0.log"
Sep 15 21:58:27.550407 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.550380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/kube-rbac-proxy/0.log"
Sep 15 21:58:27.583705 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.583675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/kube-rbac-proxy-metric/0.log"
Sep 15 21:58:27.627501 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.627427    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/prom-label-proxy/0.log"
Sep 15 21:58:27.657136 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.657104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8468bb53-031f-4260-86f5-7ac8e3974f5f/init-config-reloader/0.log"
Sep 15 21:58:27.782886 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.782856    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cddnv_61c89006-c0e2-46db-a8ab-16f7a63eeb96/kube-state-metrics/0.log"
Sep 15 21:58:27.841018 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.840947    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cddnv_61c89006-c0e2-46db-a8ab-16f7a63eeb96/kube-rbac-proxy-main/0.log"
Sep 15 21:58:27.910220 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.910126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cddnv_61c89006-c0e2-46db-a8ab-16f7a63eeb96/kube-rbac-proxy-self/0.log"
Sep 15 21:58:27.963071 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:27.963039    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-fb4bc949b-nr7xr_978ff269-e496-4abd-b062-560e64a733df/metrics-server/0.log"
Sep 15 21:58:28.024245 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.024204    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-kw422_ccdb529a-a8f1-42a8-82ac-6568c0387667/monitoring-plugin/0.log"
Sep 15 21:58:28.425143 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.425108    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ws9f2_66b1d5ff-2546-4616-a72a-0e5dacdff116/node-exporter/0.log"
Sep 15 21:58:28.475270 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.475234    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ws9f2_66b1d5ff-2546-4616-a72a-0e5dacdff116/kube-rbac-proxy/0.log"
Sep 15 21:58:28.564073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.564045    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ws9f2_66b1d5ff-2546-4616-a72a-0e5dacdff116/init-textfile/0.log"
Sep 15 21:58:28.653204 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.653121    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-gh2cd_a7ecd8f1-a2d3-414a-94eb-ea324edf341a/kube-rbac-proxy-main/0.log"
Sep 15 21:58:28.729586 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.729556    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-gh2cd_a7ecd8f1-a2d3-414a-94eb-ea324edf341a/kube-rbac-proxy-self/0.log"
Sep 15 21:58:28.796077 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.796044    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-gh2cd_a7ecd8f1-a2d3-414a-94eb-ea324edf341a/openshift-state-metrics/0.log"
Sep 15 21:58:28.870299 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.870237    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/prometheus/0.log"
Sep 15 21:58:28.945938 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:28.945907    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/config-reloader/0.log"
Sep 15 21:58:29.008071 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.008041    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/thanos-sidecar/0.log"
Sep 15 21:58:29.082140 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.082102    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/kube-rbac-proxy-web/0.log"
Sep 15 21:58:29.172044 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.172011    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/kube-rbac-proxy/0.log"
Sep 15 21:58:29.222113 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.222026    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/kube-rbac-proxy-thanos/0.log"
Sep 15 21:58:29.281231 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.281203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b95ea1dd-7a20-47c6-b614-72bb6fd3e09a/init-config-reloader/0.log"
Sep 15 21:58:29.364308 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.364272    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rhs92_f96bf148-fca4-4bb4-99eb-278326cccd89/prometheus-operator/0.log"
Sep 15 21:58:29.413366 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.413331    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rhs92_f96bf148-fca4-4bb4-99eb-278326cccd89/kube-rbac-proxy/0.log"
Sep 15 21:58:29.458941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.458911    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pkdmd_b2f4ccfa-a9dd-4447-bc0d-6cb239cb6ebc/prometheus-operator-admission-webhook/0.log"
Sep 15 21:58:29.524963 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.524885    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699449f55-t62h8_cade5465-0375-4250-9dac-31e5503d0453/telemeter-client/0.log"
Sep 15 21:58:29.571955 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.571920    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699449f55-t62h8_cade5465-0375-4250-9dac-31e5503d0453/reload/0.log"
Sep 15 21:58:29.624650 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.624622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699449f55-t62h8_cade5465-0375-4250-9dac-31e5503d0453/kube-rbac-proxy/0.log"
Sep 15 21:58:29.709147 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.709114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/thanos-query/0.log"
Sep 15 21:58:29.774385 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.774351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/kube-rbac-proxy-web/0.log"
Sep 15 21:58:29.834890 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.834810    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/kube-rbac-proxy/0.log"
Sep 15 21:58:29.891943 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.891912    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/prom-label-proxy/0.log"
Sep 15 21:58:29.952873 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:29.952840    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/kube-rbac-proxy-rules/0.log"
Sep 15 21:58:30.006567 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.006536    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b4974db4b-r7gx5_7ff809f2-20e9-45a2-bacb-b7e58f6d3a53/kube-rbac-proxy-metrics/0.log"
Sep 15 21:58:30.586067 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.586024    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"]
Sep 15 21:58:30.590774 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.590749    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.626333 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.626295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"]
Sep 15 21:58:30.714700 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.714650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxs6g\" (UniqueName: \"kubernetes.io/projected/711ab6b5-68d4-442c-b8c9-9b130a0e9259-kube-api-access-wxs6g\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.714941 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.714911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-proc\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.715001 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.714968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-lib-modules\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.715073 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.715047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-sys\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.715125 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.715091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-podres\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816302 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-proc\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-lib-modules\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816362    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-sys\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-podres\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816496 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxs6g\" (UniqueName: \"kubernetes.io/projected/711ab6b5-68d4-442c-b8c9-9b130a0e9259-kube-api-access-wxs6g\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.816926 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.816890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-proc\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.817066 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.817041    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-lib-modules\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.817485 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.817317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-podres\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.817626 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.817376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711ab6b5-68d4-442c-b8c9-9b130a0e9259-sys\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.826444 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.826412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxs6g\" (UniqueName: \"kubernetes.io/projected/711ab6b5-68d4-442c-b8c9-9b130a0e9259-kube-api-access-wxs6g\") pod \"perf-node-gather-daemonset-pskjz\" (UID: \"711ab6b5-68d4-442c-b8c9-9b130a0e9259\") " pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:30.903939 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:30.903846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:31.217034 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:31.216992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"]
Sep 15 21:58:31.222322 ip-10-0-135-105 kubenswrapper[2578]: W0915 21:58:31.222295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod711ab6b5_68d4_442c_b8c9_9b130a0e9259.slice/crio-31aaf22d96ebb19abf809993a7faa7fafebf450a1ad0da862150226ad72be617 WatchSource:0}: Error finding container 31aaf22d96ebb19abf809993a7faa7fafebf450a1ad0da862150226ad72be617: Status 404 returned error can't find the container with id 31aaf22d96ebb19abf809993a7faa7fafebf450a1ad0da862150226ad72be617
Sep 15 21:58:31.768265 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:31.768219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz" event={"ID":"711ab6b5-68d4-442c-b8c9-9b130a0e9259","Type":"ContainerStarted","Data":"aa55107ef2e087bb7ec73e7115681d71aaac86eb6469958b93dc40dbd241d234"}
Sep 15 21:58:31.768889 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:31.768863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz" event={"ID":"711ab6b5-68d4-442c-b8c9-9b130a0e9259","Type":"ContainerStarted","Data":"31aaf22d96ebb19abf809993a7faa7fafebf450a1ad0da862150226ad72be617"}
Sep 15 21:58:31.769060 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:31.769048    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:31.806785 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:31.806700    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz" podStartSLOduration=1.806679272 podStartE2EDuration="1.806679272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:31.806024758 +0000 UTC m=+2540.206998004" watchObservedRunningTime="2026-09-15 21:58:31.806679272 +0000 UTC m=+2540.207652519"
Sep 15 21:58:34.654171 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:34.654145    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dw79_4cce900c-bd00-4547-9b78-f0b5635bd252/dns/0.log"
Sep 15 21:58:34.677140 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:34.677111    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dw79_4cce900c-bd00-4547-9b78-f0b5635bd252/kube-rbac-proxy/0.log"
Sep 15 21:58:34.879594 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:34.879566    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xwltq_a31cf26a-9cfe-4fe8-9aab-dfba02d0b1d2/dns-node-resolver/0.log"
Sep 15 21:58:35.475992 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:35.475953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5ddc8578db-7l7rv_00fc94d5-a8f9-459f-8480-d19dcf8b879f/registry/0.log"
Sep 15 21:58:35.559408 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:35.559379    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v4fxj_1d421be5-c692-4b4e-9510-ad6741f05228/node-ca/0.log"
Sep 15 21:58:37.060019 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:37.059990    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-z4mdg_23a497b4-8e8b-4bf3-a03f-6af1bdd58259/kube-auth-proxy/0.log"
Sep 15 21:58:37.139794 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:37.139759    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rfw5s_13951561-032d-4591-88e5-ea2e0befd373/payload-pre-processing/0.log"
Sep 15 21:58:38.158457 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:38.158420    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lgq7k_4b2750c1-af2a-4107-bdbb-efe9eb5e8763/serve-healthcheck-canary/0.log"
Sep 15 21:58:38.791082 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:38.791046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j7p98/perf-node-gather-daemonset-pskjz"
Sep 15 21:58:38.979182 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:38.979155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7rm_49f3aed7-634d-4cac-885e-0242bf955c33/kube-rbac-proxy/0.log"
Sep 15 21:58:39.002937 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:39.002905    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7rm_49f3aed7-634d-4cac-885e-0242bf955c33/exporter/0.log"
Sep 15 21:58:39.029438 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:39.029405    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7rm_49f3aed7-634d-4cac-885e-0242bf955c33/extractor/0.log"
Sep 15 21:58:42.055120 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:42.055087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-z4mdg_23a497b4-8e8b-4bf3-a03f-6af1bdd58259/kube-auth-proxy/0.log"
Sep 15 21:58:42.148349 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:42.148322    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rfw5s_13951561-032d-4591-88e5-ea2e0befd373/payload-pre-processing/0.log"
Sep 15 21:58:42.297313 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:42.297199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bd4b7887f-qhsnm_40d99f10-e736-4ba1-a2d9-173941e51abd/manager/0.log"
Sep 15 21:58:42.490276 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:42.490245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-f6zdk_27835161-a6a8-4f98-bc79-8825748e6b66/manager/0.log"
Sep 15 21:58:42.518158 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:42.518133    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596bc79d9b-f6zdk_27835161-a6a8-4f98-bc79-8825748e6b66/copy-manifests/0.log"
Sep 15 21:58:44.053190 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:44.053155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-dccb565dd-96nhw_d66337a6-479c-4a17-bf61-960409c179de/manager/0.log"
Sep 15 21:58:44.297001 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:44.296932    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58bcc99dc6-z4mdg_23a497b4-8e8b-4bf3-a03f-6af1bdd58259/kube-auth-proxy/0.log"
Sep 15 21:58:44.352240 ip-10-0-135-105 kubenswrapper[2578]: I0915 21:58:44.352211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-rfw5s_13951561-032d-4591-88e5-ea2e0befd373/payload-pre-processing/0.log"